Determining position or orientation of object in three dimensions

ABSTRACT

Methods for assembling, handling, and fabricating are disclosed in which targets are used on objects. The targets can be specifically applied to the object, or can be an otherwise normal feature of the object. Conveniently, the targets are removable from the object or covered by another object during an assembling process. One or more robots and imaging devices for the targets are used. The robots can be used to handle or assemble a part, or a fixture may be used in conjunction with the robots. Conveniently, the CAD design system is used in designing the targets as well as for the assembly process using the targets. A plurality of targets can also be used to monitor and inspect a forming process such as on a sheet panel.

This application is a division of application Ser. No. 07/875,282, filedon Apr. 29, 1992 which was a continuation of application Ser. No.07/478,078, filed Feb. 9, 1990, now U.S. Pat. No. 5,148,591, which was acontinuation of application Ser. No. 07/110,541, filed Oct. 20,1987, nowabandoned, which was a continuation of application Ser. No. 06/865,637,filed May 14, 1986, now abandoned, which was a continuation ofapplication Ser. No. 06/660,280, filed Oct. 12, 1984, now abandoned,which was a continuation-in-part of application Ser. No. 06/348,803,filed Feb. 16, 1982, now abandoned, and a continuation-in-part ofapplication Ser. No. 06/453,910, filed Dec. 28,1982, now abandoned, anda continuation-in-part of application Ser. No. 06/323,395, filed Nov.20, 1981, now U.S. Pat. No. 4,482,960, and a continuation-in-part ofapplication Ser. No. 06/651,325, filed Sep. 17, 1984, now U.S. Pat. No.4,769,700, and a continuation-in-part of application Ser. No.06/592,443, filed Mar. 22,1984, now U.S. Pat. No. 4,602,163, which was acontinuation-in-part of application Ser. No. 06/262,492, filed May 17,1981, now U.S. Pat. No. 4,453,085.

BACKGROUND OF THE INVENTION

The following are applications in the same field showing the state ofthe art which are herein incorporated by reference:

1. Robot Calibration, U.S. Ser. No. 06/453,910, now abandoned.

2. Electro-Optical Systems for Control of Robots, Manipulator Arms andCoordinate Measuring Machines, or “Robots and Manipulator Arms”, U.S.Ser. No. 592,443, filed Mar. 22,1984, now U.S. Pat. No. 4,602,163.

3. Robot Tractors U.S. Ser. No. 06/323,395, now U.S. Pat. No. 4,482,960.

4. Robot Tractors, Vehicles and Machinery, U.S Ser. No. 651,325, filedSep. 17,1984, now U.S. Pat. No. 4,769,700.

5. Electro-optical sensor systems for thread and hole inspection U.S.Ser. No. 06/064,867, now U.S. Pat. No. 4,315,688.

6. Method and apparatus electro-optically determining the dimension,attitude and location of objects: U.S. Ser. No. 34,278.

7. Method and apparatus for determining physical characteristics ofobject and object surfaces: U.S. Ser. No. 06/015,792, now U.S. Pat. No.4,373,804.

8. New photodetector array based optical measurement systems: U.S. Ser.No. 06/163,290, now U.S. Pat. No. 4,394,683.

9. Electro-optical inspection, U.S. Ser. No. 06/073,226, now abandoned.

10. Co-ordinate measuring method and device, U.S. Ser. No. 06/201,081,now abandoned.

11. Electro-optical sensors with fiber optic bundles, U.S. Ser. No.06/173,370, now U.S. Pat. No. 4,441,817.

12. Electro-optical surface roughness measurement and control U.S. Ser.No. 06/240,459, now abandoned.

13. Apparatus for determining dimensions, U.S. Ser. No. 06/134,465, nowU.S. Pat. No. 4,403,860.

14. High speed electro-optical inspection, U.S. Ser. No. 06/203,866, nowabandoned.

15. Fiber optic based robot controls, U.S. Ser. No. 06/200,401, now U.S.Pat. No. 4,460,826.

16. Electro-optical sensors for machine tool and robotic inspection,U.S. Ser. No. 06/247,399, now abandoned.

17. Electro-optical systems for control of robots, manipulator arms andcoordinate measurement machines U.S. Ser. No. 06/262,497, now U.S. Pat.No. 4,453,085.

18. Method and apparatus for determining wear or breakage of tools andother defects, U.S. Ser. No. 06/323,397, now U.S. Pat. No. 4,420,253.

19. Electro-optical systems for detection of leakage and blockage, U.S.Ser. No. 06/323,399, now abandoned.

20. Productivity improvement via robotic electro-optical part and toolinspection, U.S. Ser. No. 06/323,396, now abandoned.

21. Robot tractors, U.S. Ser. No. 06/323,395, now U.S. Pat. No.4,482,960.

22. Method and apparatus for determining physical characteristics ofobject outer surfaces U.S. Ser. No. 06/015,614, now U.S. Pat. No.4,326,808.

23. Method and apparatus for determining dimensional informationconcerning an object, U.S. Ser. No. 06/234,729, now U.S. Pat. No.4,465,374 (division of U.S. Ser. No. 06/015,792, now U.S. Pat.No.4,305,661).

24. Method and apparatus for detection of surface deformities (divisionof Ser. No. 06/15,792) U.S. Ser. No. 06/234,728, now abandoned.

25. “Linear” continuation of U.S. Ser. No. 06/015,792, now U.S. Pat. No.4,506,980.

26. “Circular” continuation of U.S. Ser. No. 06/015,792, now U.S. Pat.No. 4,465,374.

27. Optically controlled plumbing apparatus U.S. Ser. No. 06/029,840,now abandoned.

28. Optically controlled bathing systems U.S. Ser. No. 06/023,150, nowabandoned.

29. Electro-optical and robotic casting quality assurance, U.S. Ser. No.06/273,385, now U.S. Pat. No. 4,409,718.

30. Controlled machining of combustion chambers, gears and othersurfaces including methods for obtaining correct combustion chambervolume in finished engine assemblies, U.S. Ser. No. 06/238,702, now U.S.Pat. No. 4,559,684.

Flexible robot assembly is often very difficult in the absence ofmachine vision sensors to guide the operation. Even with such sensors,operation must be both accurate, ultra reliable, and fast enough to bejustifiable relative to human labor. These criteria are seldom met bypresent day vision systems employing arbitrary gray level images and thelike.

The target based invention described in reference 1 above has wideapplication to the assembly process. Described therein are severalembodiments illustrating target based techniques which can overcome thelimitations of present systems. The key to the use of the disclosedsystems is that the target points on the part are easily discernable andunambiguous, after processing using rapid devices and other high speedanalysis software.

The target system functions well because it is based on high contrastimages and mathematical equations. To use targets one must know the partfeature data base relative to the target points on the part. Targets ontooling, pallets and fixed members may also be of use. Special retroreflective targets give best contrast, but targets can be holes, comersor other easily determined natural part features.

Finally, where special targets are used which would not normally bepresent, techniques are disclosed to make these unobtrusive.

SUMMARY OF THE INVENTION

In accordance with the present invention, a method for controlling anassembly process is provided in which at least one first robot holds afirst part in relation to a second part. Targets are provided on atleast one of the robots or the first or second part. A TV camera thendetermines the position of the targets. From this determination, theassembly process is controlled.

In a preferred embodiment, the part held by the robot has a face onwhich the targets are located such that the TV camera can view thesetargets. In this manner, the robot is guided to handle or assemble thepart.

A method for fabrication or assembly in which a fixture is provided isalso disclosed. The location of a part having targets is determined witha TV camera and a robot then places the part on the fixture dependingupon this determined location. A further object may be located on thefixture.

During a dynamic fabrication, it is also possible with the presentinvention to target the part so that corrections can be made duringprocessing.

Preferably, the targets are located in a finished assembly in such amanner as to be out of view. These targets can be covered up by a secondobject assembled with respect to the first object if applicable.

The present invention also includes a method for guiding objects intoopenings or onto protrusions. A plurality of target features areinitially provided adjacent an opening or protrusion. These targetfeatures are imaged with a TV camera in order to determine the locationof the target features. From this determined target location, an objectis guided onto the opening or protrusion.

The present invention also makes use of a CAD system in a method offabrication. Initially, the design of a finished assembly of parts isprovided in a CAD system. The CAD system is also provided withindividual part designs which are used to make up the assembly. Thesedesigns include special targets or natural features which serve astargets. The entire system is then provided with a logical progressionof functions feeding program/functions to one or more robotic automationdevices for assembly. Assembly is assisted by the use of TV cameraswhich are provided with the design data. The parts are then suitablyassembled using images of the targets determined by the camera.Initially, the CAD system can be used to simulate the fabricationprocess.

In another preferred embodiment of the present invention, targets areprovided on an object before a forming operation is performed. Thelocation of the targets after forming are then determined. From thisdetermined location, the attitude, shape, or dimension of the object orportions thereof are also determined. Depending upon the determineddata, the handling, assembling, inspecting, or working the object isthen effected.

This invention further relates to methods and apparatus for determiningthe position of an object and guiding robots or other automation tohandle or work on said object.

There are many instances in which it is desired to know the position ofan object. In the manufacturing field, such instances include theposition of objects along mass production lines, particularly thosewhich are highly automated. For example, in a mass production line, itis frequently necessary to know with considerable precision the positionof an object suspended from a conveyor system. This is particularly truein automated systems involving the use of robots for it is fundamentallynecessary that the object be in a known position, relative to the robot,before the robot can execute a desired manipulation of the object.

In some instances, mechanical means are used to position the objectand/or to orient the object in proper position for automatedmanipulation. For many applications, however, it is necessary to provide“vision” in order to determine the position of an object. This isparticularly true in the case of robots which can perform a myriad ofphysical manipulations, all automatically. It is well recognized in thefield of robots that robotic “vision” is one of the major obstacles tomuch wider use of robots which are presently quite sophisticated interms of the manipulations of which they are capable.

Specifically the problem in plants of operationally using robots tohandle or work on random parts on continuous conveyors is an enormousone. Since such conveyors are omni present in plants of all types, thisproblem must be effectively dealt with, if large scale robot usage is tobecome a reality.

In doing so, there are many types of electro-optically based “machinevision” systems which can be utilized. Historically, these systems havebeen based on reflective viewing of objects through their gray scalelevels which poses extremely difficult problems. The trend is thence toever more complex systems, which runs counter to good plant reliability.

This inventor, for example, has been involved in the installation ofnearly 1,000 electro-optical sensor units in plants of varying types forinspection. Substantial difficulties have been encountered when suchelectro-optical image based sensors were utilized to obtain part images,particularly in reflection.

When one considers that the robot based system must achieve areliability far higher than even these inspection based units, in orderthat it not ruin the product, drop it on the floor etc., it becomesapparent that a simple and reliable means of solving these problems isrequired.

This invention seeks to illustrate such means. In particular, solutionis possible if one restricts the problem simply enough to targetedobjects. This then leads to the possibility of tracking the parts or thecontainers, conveyors etc. so targeted, possibly using further sensorsif required to find parts within these containers, instrumented grippersor the like.

The application by the inventor, U.S. Ser. No. 200,401, illustrated inthe embodiment of FIG. 13, instrumented monorail and walking beamconveyors utilizing fiber optics directed through portions of theconveyor apparatus which could be illuminated on demand in order toprovide one or more targets for tracking or homing purposes usingrobotic or other automation.

Also described in U.S. Ser. No. 06/200,401 (now U.S. Pat. No. 4,460,826)are many other novel features of interest. These are:

(a) The general concept of use of such ‘active’ lighting in automationand particularly the use of fibers therefore.

(b) The use of ‘active detection’ wherein the light is directed from therobot into one end of a fiber, and sensed at the opposite end of saidfiber.

(c) The use of other materials than fibers, for example, translucentfixtures of Teflon or ceramic.

(d) The use of multiple target points on the illuminated piece to betracked.

(e) The use of blow-offs to keep the targeted fixtures clean.

(f) The use of pulsed or modulated light sources discrimination againstbackground noise.

(g) The use of light sources and electro-optical sensors both located onrobot where the light source of the robot is directed to a predictedentrance point of the fiber(s) and the light emanating from the oppositeend of the fiber(s) is sensed by the camera of the robot. Two robots forexample could be used, one to light the part or fixture, the other tosense it.

In addition, the application described how to track conveyors carryingparts and made reference to the tracking the parts themselves.

It is an object of the present invention to provide a method andapparatus for determining the position of a targeted object or objectcarrier. It is a further object of the invention to provide such methodand apparatus having particular suitability for providing robotic“vision” and to disclose practical system based thereon.

It is further intent of this invention to expand the previouslydisclosed concepts beyond simply the fixtures themselves and the partswithin the fixture, to the targeting and identifying of containers andparts of all descriptions, for example to baskets, trays, cartons, toolsfor pickup, parts in warehouse bins, and indeed the parts themselves.

In addition, this invention elaborates further on the use of othertargeting materials than fiber optics, not only transmissive materialsof other sorts, but objects such as glass beads, drilled fascets,casting risers and the like.

It is a further purpose of this invention to show other means in thesensing which can give improved target position and data. Particularlyof interest are those related to apparatus such as Pinkney, U.S. Pat.No. 4,219,847.

It is further a purpose of this invention to show specifically howcertain parts in overhead monorail conveyors can be picked off underrandom situations with the high reliability needed to work in plants.

It is further a purpose of this invention to show means for outliningthe edges of targets for use with stereo cameras or other sensing meansnot necessarily based on point targets.

It is a further purpose of this invention to show means for coding thevarious fiber input or outputs, or other targets by use of colors ormodulation frequencies. This is also possible with inserted glass beadsand retro-reflectors.

Finally, it is a desirable purpose of this invention to show thatstandardized systems, based on such tracking, can be used across thetotal spectrum of manufacturing and industry with very little change aslong as one sticks to certain target principles. This allows the widespread use of reliable guide robots at affordable cost.

Other features and advantages of the present invention are stated orapparent from a detailed description of presently preferred embodimentsof the invention found hereinbelow.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates a targeted assembly fixture with partloading by a robot.

FIG. 2 is a simulation block diagram.

FIGS. 3a and 3 b schematically illustrate a dual robot version of theinvention, with specialized target concepts including targets on twosides of parts, on end effectors and the like.

FIGS. 4a and 4 b schematically illustrate, respectively, an assemblytechnique using targets which surround an opening and targets which arecovered up by other parts or assembled to point the targets away fromview.

FIG. 5 schematically illustrates transformation of target data bases dueto forming.

FIG. 6 schematically illustrates flexible “fixtureless” roboticassembly.

FIG. 7 schematically illustrates an assembly cell according to theinvention.

FIG. 8 is a diagrammatic side elevation view of an overhead conveyorsystem utilizing the present invention,

FIG. 9 is an enlarged diagrammatic perspective view of hook 14 of FIG.1.

FIG. 10A, 10B illustrates a complete system for the picking oftransmission clutch parts off of overhead monorail conveyors in anautomatic transmission plant. The sensor is of the general typedescribed by Pinkney, utilized to track the conveyor carrier(s) whichare targeted by means herein disclosed. Optionally, an additional sensoror analysis means on the robot may be used to find the part within thecarrier.

FIG. 11 illustrates target embodiments on the carriers used in FIGS.10A, 10B.

FIGS. 12A, 12B illustrates an embodiment showing methods of targeting aplastic door panel having built-in optical fibers.

FIG. 13 illustrates a car windshield with lossy fibers to outline itsperiphery, which is imaged by stereoscopic cameras.

FIGS. 14A, 14B illustrates a part targeting embodiment employing drilledholes.

FIGS. 15A, 15B, 15C illustrates a part targeting embodiment wherein saidtargets are cast into the part or are appendages thereto.

FIG. 16 illustrates a part targeting embodiment wherein said targetscomprise directional or color reflective elements which may be molded orpressed in.

FIG. 17 illustrates a robotic system employing targeted boxes randomlyspaced on a roller conveyor, utilizing either targets printed onto thebox, fibers, or retro-reflectors.

FIG. 18 illustrates means for targeting a tool.

FIG. 19 illustrates alternative means for creating sitable targets, anduse on car body assembly.

FIG. 20 illustrates reusable targets for parts.

FIG. 21 illustrates a method of assembling cars according to theinvention.

FIG. 22 illustrates a sensor embodiment according to the invention.

FIGS. 23A, 23B, 23C and 24 illustrate further part targetingembodiments.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 illustrates an assembly operation using a robot 10 with a controlcomputer 11 to place a part 12 of a car on an assembly fixture 15 whichhas been targeted according to the invention. As is desirable, the CADsystem 30 (which has created the design for both the part and/or thefixture) has recorded the target data points 40, 41, 42 and 43 that areto be used by a camera unit 35 to locate the part 12 and/or the featuresof the part 12 that have been designated as targets plus the targetpoints of the assembly fixture. It is noted that in some cases assemblyfixtures are not required as pointed out below. When this data isobtained, the CAO system 30 downloads to the robot control 11 thedescription of the target point locations and the robot then homes in onthese points using the present invention.

In a typical example, a hood inner support is part 12. Part 12 is loadedin a programmable manner from the download data onto locators 17, 18, 19and 20 on a fixture 15 where targets 35, 37, 38 and 39 have been placed.Targets 36, 37, 38 and 39 are either on or near the locators 17, 18, 19and 20, or on points 50, 51, 52 and 53 around the framework of thefixture 15 which are related to the locators 17, 18, 19 and 20. Thecamera system 35 located on the robot (or alternatively off the robot asshown) locates these points and guides the robot 10 in. Differentialclosing data between part and locators can be provided to allow a smoothclose and mate operation.

To assemble hood outer panel 60 to the inner part 12, adhesive forexample, is first applied to hood inner part 12. The adhesive isconveniently applied using the same robot 10 and using the fixture orpart target data to guide the robot 10 (although once the part 12 iswell-located on the fixture 15, it is known accurately enough inposition to not require additional vision sensing except as averification check).

The robot 10 goes and gets the hood outer panel located in rack 61 usingeither of two methods. First, it can locate the hood by its corner edges70, 71, 72 and 73 as target points, grasp the outer panel 60 with asuction cup, and place it on the inner support part 12.

Another method is that the robot 10 grasps the panel 60 arbitrarily atthe best estimated location on its outer surface. The robot 12 thenpresents the rear of the surface of panel 60 having holes 80, 81, 82, 83to camera 35. Camera 35 then establishes the part reference to the robotso that the hood outer panel 60 can be reliably placed over the innerpart 12 and joined.

The fixture design and the target relationships to the part design canbe optimally matched in the design computer 100 as shown in FIG. 2. Thisallows a simulation routine 102 of the whole process at a point beforeconstruction. Unlike other hypothesized robot simulation activities,this one is based on the sure knowledge that the target points will beidentifiable within any given practical accuracy range because onlyhighly well defined targets are utilized which are at sufficientlyprecise locations to the data bases of the part or fixture to assuresuccess.

As shown in FIG. 2, simulation routine 102 begins with design part box104 which takes account of the design data of the part, including anyspecial points (if any). Next, from this data, target points aredesignated on the part as shown in box 106. The operational parametersare then specified as indicated by box 108. These parameters includecamera-processor types, type of robot(s), distances, range of possibleorientations, tracking algorithms used, photogrammetric algorithms used,special target indicators, and the like. Using the operationalparameters selected, an operation such as assembling or handling istried as shown by box 110. The results of this “try” are then determinedas indicated by box 112. These results are such things as cycle time,functionality, etc. Depending on the results obtained, variousparameters are varied in an effort to improve the process as shown bybox 114. Among the parameters which might be varied are the following:change equations, redesign part, change target points, apply specialtargets, add light sources, and add front end processor.

After the appropriate parameters are varied, the operation is againtried as indicated by box 116. If not successful, the parameters areagain varied and the operation rerun until a successful operation isachieved. At this point, the operation is ready to be integrated withother operations as indicated by box 118. It should be noted that CADsystem 100 would have ideally programmed the parameter (speed, accuracy,risk etc.) of various target choices to be seen using various cameralighting, image processing, and the like.

In addition, simulation 102 can continually optimize the solutions ofthe photogrammetric target equations to allow the best performanceaccuracies to be obtained within the desired cycle time of theoperation. For example, the target choice and location, number oftargets etc., can be changed on a CAD system and operation of theprocess using robots interacting with these targets simulated andphotogrammetric and/or tracking equations optimized.

A simulation block diagram is shown in FIG. 2. With the system, itbecomes cost effective to develop simulation programs for robot systemssince one has an assurety of vision based guidance. This is not truewith other guidance systems in operation today, and particularly not sowith those capable of the high speed accurate operation required tojustify cost in modern manufacturing.

FIG. 3a illustrates a variation on this particular embodiment in which adual set of robots is used. In this case, Robot A picks up the part 300and presents it to Robot B which then inserts a shaft part 301 into asleeve 302 in part 300. This assembly operation could be any other typeof operation as well.

Part 300 is picked up by the robot gripper 304, preferably, but notnecessarily, using target points 310, 311, and 312 on part 300 viewed bycamera 321. When it is presented to part 301, the second robot caninteract in any one of several ways. In the first case, the part 300 hastarget points that can be identified such as holes or other purposelyplaced targets that can be identified relative to it. In this case, thecamera unit 320 on Robot B can home in on those points. Again, asbefore, all of this data can be downloaded from a central computer 350.

Another possibility is to target the Robot A end effector or tooling 304holding the part 300, as shown with targets 340 and 341. Since part 300has been picked up originally in an accurate manner relative to thetooling 304, the tooling 304 then can act as the data base to providethe target data points for the second Robot B. This is particularly truesince these target data points can be idealized targets comprising retroreflective material or active fiber illumination which are almostperfect in the ability to be accurately judged. Thus, minimal stackuperror due to this transfer of location data occurs.

It is noted that the camera 321 on Robot A can “home-in” on targetedtooling plate 360 on Robot B having 3 targets as shown while the cameraon Robot B can look at tooling 304 on robot A. Each one can thus home inon the other.

This ability of Robots A and B to look at each other is illustrated inFIG. 3b. A targeted plate 361 surrounding camera 360 is attached torobot arm 362, while a similar targeted plate 371 is adjacent camera 370on robot arm 372.

Note that robot arm 372 could, for example, be instead a fixed structuresuch as a drill or router unit with camera 360 (or 370) homing the robotarm 362 in on it with a part in its gripper using targeted plate 371 (or361).

A third possibility is to target both robots in a manner visible fromabove (or elsewhere) and use an overhead camera system 390 to monitorthe closing target conditions of one to the other. This is schematicallyshown in FIG. 3a.

Another version is a front/back target situation where control transferis made between target sets on a part. A part can be picked up (say byrobot A) based on targets on one side, and presented to a camera onrobot B whose camera locks in on targets on the other side.

FIG. 4a illustrates a robotic arrangement for putting parts into holesor other openings. The area around the hole 500 in part 499 is targetedwith targets 501, 502, 503 and 504. Typically 3 or more targets areutilized to allow the total 6 degree of freedom approach to the holes tobe made. In this example, the camera 510 on robot 530 senses the targetpoints 501, 502, 503 and 504, and knowing their locational relationshipsto the hole 500 and its centerline, guides the part 520 such as a pininto hole 500.

TV camera 510 can be located on the robot arm, or elsewhere. The camerasensor unit may also be augmented by triangulation sensors using a diodelaser or other light source 540 to project the laser spot 541 or otherzones onto the part to further aid the guidance. Such spots or zones canalso be viewed by the same camera 510.

The targets around the hole can be anything suitable. For example, on acast iron block or a cast aluminum block, these can be cast inindentations, raised areas, etc. perhaps with special signatures (e.g.triangles, stars, etc.) to allow their easy acquisition. A variety ofother possibilities are also in order. The lighting for such targets isusually provided by a source on the robot arm although the maximumcontrast, indented, or raised targets should be sometimes lit at anangle (see copending application reference 1 on lighting for robotapplications.)

The particular opening can be any opening, not just a hole. For example,the points at which a gas tank lid is to be placed onto the side of acar rear quarter panel could be used. In this case, the target pointsmight be painted onto the body side outer surface, or other innerpoints, which would be covered up by the lid.

In this general case, the target points on the surface are covered upafter the part is assembled on the shaft by the part. This way they areremoved from aesthetic considerations. There are numerous examples wherethe object involved can be used to cover up the target points that werefirst used to find the 6 degree of freedom or multi degree of freedomlocation of the object.

For example, consider FIG. 4a where the part could be a radio knob witha disc 598 (dotted lines) on a shaft 520. When assembled into hole 501in dashboard 499, disc 598 covers up targets 500-504.

It is noted that hole 500 in FIG. 4a could also be a protrusion like ashaft, stud, etc., assembly to which also requires multi degree offreedom guidance.

Target points can be located on the end of protruding objects as well.For example, it is often necessary to insert something with multipleholes onto multiple pins (or vice versa). This is the case of wheelbolts on a car. In this case, illustrated in FIG. 4b, the targets arethe ends of the wheel bolts 601, 602, 603 and 604 which are suitablytargeted according to the invention, for example, with reflective paint.Conveniently, the camera 620 can be mounted in the tooling 621 holdingthe wheel 622 so as to look through the wheel at the bolt ends.

Another example of targets which can be camouflaged is the use oftargets on the back side of an object as was illustrated in FIG. 1. Inthis case, it is desired for example to assemble a part which has beentargeted for ease of manipulation to another part. The targets in thiscase are purposely chosen to be on the inside when the assembly iscomplete.

For example, let us consider a process where a robot utilized forassembling a house is used. The robot picks up targeted board lumberwhich is targeted on its back face with the target points, or what willbe the back face in the home. The back face is then presented by thisfirst robot to a second robot which nails the board to other boards thathave already been assembled. The assembly is done in such a manner thatthe target points that have been used are mounted on the inside. Indeed,this works out quite well since the robot can also look at target pointson the outside of the rest of the structure and determine from them thelocation to place this first part over same. As the structure is totallyassembled, all targets end up pointing inside (or are covered up). Thisallows the use of such targets without creating aesthetic problems onthe outside.

This coverup was shown relative to the ratio knob in FIG. 4a but alsowould be the case here on the inside where one would cover it up laterwith wallboard.

Consider now FIG. 5 which illustrates the use of target points 700 allover a structure 702 and particularly on formed parts. The intent hereis twofold. The first is to deduce from the changes in the targetpattern the effects of forming on the part 702. For example, it may bedesired to obtain the new dimensional shape of the part 702 using thedata base now resulting from the forming process 706 and any strains orany in-plane deflections which may have occurred as a result of theforming or welding process, etc. This can be used to control the process706 as well as feed data forward as to the new shape for further roboticassembly.

For example, consider the case of metal body panels or aircraft panels.They start out as steel blanks 702 which can be imprinted with a dottarget pattern on one inch centers throughout the total width. Thetargets 700 can be rather precisely applied, or if not, can be initiallymonitored by a camera means 708 to discern what the pattern is. Suchapplications could be through a roll coat 704, an etch process, or thelike.

This sheet of steel 702 is then formed into a new form such as a cardoor or, in a more simpler case, a deck lid. It is now desirable to suethose target points 700 as viewed by camera means 710 which have nowchanged their form to determine any irregularities of the formingprocess as well as to establish the new data base for the part. Wherethis data base (namely shape and so forth) is a known desired one as itis in the case of a door, one would then like to compare the desireddata with that resulting from the new pattern. Furthermore, this newdata base can be utilized in the actual handling of the part betweenstations of a press line, for example, as long as it is known what toanticipate in terms of what it is. Indeed, it is noted that rather thanputting down a roll coated impression every one inch at squares on theinitially formed blank, one can actually roll coat down a series of dotsof other target points which make a square pattern when optimallyformed.

Now, take this case one step further, where a formed part 702 such as adeck lid has dot points 700 on it. This formed part is then to have apart 712 welded to its rear by spot weld robot 716, which part 712 isthe inner stiffener of the deck lid. The dots 700 are viewed by a camerameans 714 and not only used in the handling process of handler 718 tomove this part 712 into place also using camera means 714, but also toposition the welded inner part 712 relative to part 702. These dots arefurthermore used to check after welding whether or not certaindistortions have occurred beyond limits in the welding process and toestablish the new data base for the part.

If new data base is satisfactory, the part is moved on by a suitablecamera equipped handling robot to shipping or to the body build process.In this process, this formed part is mated up with other parts and againthe target points 700 can be used in this mating, for example, inputting the deck lid into a deck lid opening.

In the case of doors, it is particularly important because the door isloaded with hardware and then hung from hinges which can tend to furtherdistort the door. Indeed, if all the formed parts were made up andtargeted in such a manner, it would make the job much easier for theautomation and inspection through the total build of the part. Thisconcept goes well beyond formed parts, to all types of things. However,formed parts are the ones in which the most distortion can occur.

In the final operations, the etched on or painted inked on marks, etc.,if on the outside of the car, are generally painted over or otherwiseremoved. Naturally when such marks occur on the inner portions of themetal, no aesthetic problem exists.

This process is particularly applicable to formed metal which can springback and generally has a relatively unknown shape such that errors intarget point locations are not severe relative to the accuracy of themetal.

Another process that is somewhat similar is casting which is arelatively loose process on the same tolerance level as the forming.Plastic molding is another with the same tolerance but unfortunatelythis generally does not lead to distortion except during perhaps gluingoperations. In any of these cases, suitable target points can be moldedin, cast in, etched on, burned on as with lasers, or any suitable meanscan be utilized.

One can also use an absolute encoded target pattern, one which has noambiguity as to where each target is. This could be with clusters oftargets on different centers or patterns, different shaped targets, orthe like.

It has been noted that this process is most applicable where moreoperations occur to a given part that can change its shape and where thepart is more likely to change shape as a result of the operation (as inspringback). This means that the roll forming, hydroforming, pressforming, press brake forming and other forming operations done in theautomobile, aircraft, and offroad vehicle industry, particularly arenatural users of such technology. By judicious choice of targets andtheir applications, one can utilize these targets throughout the processfor material handling assembly and inspection purposes and generallyeffect large cost reductions in the manufacturing as well as improvegreatly the flexibility with which such manufacturing can occur.

The target application techniques can be whatever is suitable. As isobvious, the closer to a flat blank that the targets can be applied in,the better from the purpose of target application as long as thosetargets through the forming process can maintain their legibility.

Speaking of legibility, the typical technique is to utilize a solidstate TV camera to look at the target points This has provedsatisfactory, for example, to look at etched lines on die blanks inpress forming.

While target points on the formed materials and other shapes areconsidered here mainly in the light of formed steel stampings, aluminumgraphite epoxy and other formed parts, one can actually extrapolate thisto all types of structures and parts such as, for example: plywood onbuildings, concrete which sets and takes a shape and then might later beused for further operations, ground (earth) as in targeting points onthe ground which would then be used later on to guide particular itemseven though the ground had shifted or was distorted by working aroundit, and the like. Indeed, it is the general case that if one can gettargets onto the surface of structural parts or other objects in theiroriginal form before any further work and if those targets stay on thoseparts throughout the work, then they can be used the maximum number oftimes and therefore increase the cost justification of the targetingconcept. For targeting on concrete, targets such as glass beads, dottargets, or cats eye retro reflectors could be actually inserted intothe material where they would harden therein and still be visible fromthe outside. These, of course, are idealized targets. Others wouldsimply be little dots, points, or even non-circular targets. In fact,non circular ones could be advantageous from the point of view ofabsolute target coding. However, in general, circular targets aredesirable.

One must consider the integration of all these target concepts. Forexample, discussed above is targeting the points in their bare form andlooking at a changing data base for the part as it goes through thevarious forming and joining operations. Also discussed was designing thepart in the computer with the targets present thereon, the targets beingin this case either special targets or normally appearing portions ofthe part or both on any one part or on any grouping of parts.

In other copending applications such as those referenced above in theBackground section, other systems for targeting the work area of robotsand automation of guiding vehicles and other target functions areperformed. The operation too of the various vehicles and robots can besimulated in the computer relative to these target points as well.Again, a key item is that with target points one is much more assured ofaccurate reliable operation and therefore the justification for suchsimulation and expense thereof is paid back by the surety of thefunction of the final simulated structure. And too, one is dealing withmathematical representations of target points and photogrammetricequations, not relatively unknown functions of vision systems with graylevel images of object features. Simulation of dynamic target trackingalso become feasible as the problem is much more defined than with graylevel scene analysis. This also leads to faster assembly and morejustification.

One can indeed state that the whole work area and work environment ofautomation in the factory of the future could well be targeted. All theparts, the fixtures, the jigs, the tooling, the robot worked areas, thepassageways and areas of travel of automated guided vehicles carryingthe parts, the transfer conveyers, even belts on which the parts ridecould be targeted. With such target data stored in the data base of amaster control computer for any one cell or an overall host computer,one has complete control of the environment. Everything is knownrelative to known target points such as: the location of the automation,part, the part shape, material handling, bringing the part and taking itaway, relationship of other parts to the part, and the like. The datarelative to this information can be manipulated in the computer andoptimum operations derived such as trajectories of robots, forming, andthe like. It is considered within the purview of this invention tointegrate these functions to the maximum extent possible to whereflexibility is attained while still having complete control over theoperation.

FIG. 6 illustrates robotic fixtureless assembly using targets andfurther illustrates the concept of dynamic target location to adjustholding or welding robots dynamically under target control in responseto forces (clamping, welding) tending to change the part from itscorrect shape.

As shown, car cowl 802 is to be welded to flange 803 of underbody 801,under control of at least one target sensing camera 800. Camera 800senses various targets 804 on cowl 802 and targets 809 on underbody 801itself. The sensed data is then fed to computer 808 which is connectedto holding robot 805 and welding robot 810.

Robot 805 has gripper 806 holding cowl 802 in place and having targetplate 807 observable by camera 800. Robot 810 has spot weld gun 851putting flange to cowl welds on, with target plate 850 also observableby camera 800.

The holding robot 805 is moved into position to the correct locationdetermined by cowl 802 and underbody target location and the weldcommenced. If distortions occur moving any part to a wrong position, theholding robot 805 and welding robot 810 acting in concert under controlof computer 808 move to correct them. Typically, a plurality of weldingand holding robots would be utilized. Some non-adjustable ornon-programmable clamping or welding could also be used in conjunction.

The target's data base provides reliable accurate control over thisoperation and allows instant feedback correction. The procedure isflexible and could be programmed for any operation or car line.Typically, several cameras looking from different views would berequired on a complex structure.

Note that when such a fabrication cell or an assembly cell is fixed allin one spot (no moving conveyors), the camera computer 808 can memorizethe position of each of the pieces in build up, from its targets, eventhough they are later covered or partially covered up. For example, whencowl 802 is positioned to underbody 801, computer 808 records theposition of underbody 801. Then, after cowl 802 has been welded tounderbody 801, computer 808 again records its position. As a next step,computer 808 could guide A-pillars in each side from their targetsrelative to the recorded cowl position (and if desired, consideringunderbody 801, too).

For example, one might purposely slightly change the A-pillar locationsrelative to the cowl if another body portion mating to the A-pillar wasrelated more to underbody position. This avoids stackup of errors. Thisreally is a major advance in that it accounts for weld distortion-youcan measure and correct while welding and after weld you know the finallocation.

Correction of distortion or stack up of tolerances in real time isabsolutely a major advance. This would apply to adhesive bonding,riveting and other joining processes as well.

The term “TV Camera” in the invention generally connotes a dimensionallystable type, such as a solid state TV camera, e.g. a GE TN2500. However,other means for electronic imaging can be used such as Vidicons, imagedisectors, linear arrays and the like. Indeed, a mechanically scannedlaser beam, like a flying spot scanner, can be used to sweep throughspace in order to find target points some of which envisioned could beparticularly reflective or responsive to laser light. The primary goalis to find target points or features. However, the TV cameras also allownormal gray level part images to be obtained, a big secondary advantagesince this is often combined with the target based guidance function forinspection, recognition or other purposes.

A suitable target sensing vision system is that described in the priorcopending applications noted above, and it is noted that the designprogram on the computer aided design system should be able to design thetype of location of these targets which are suitable for the visionsystem.

The laying in of the targets could be by specifying operations which putthe targets on as special items, i.e. by spraying them on, sticking themon, or drilling special holes or the like. The other idea is to use theCAD system to design the part, and then to look at the part as designedin its functional way. Drawing upon the knowledge of what can constitutea natural target (as in a naturally occurring hole or corner), suitabletargets are specified. A computer program can be used, if desired, toaugment the decision as to which natural targets are the targets to beused.

The beauty of the computer design system then would be to allow you to“try out” the design with a hypothetical target set using a typicalsimulation program for a RPS (or other target determining camera system)equipped robot.

Suppose one would wish to design an operation where a sheet metal doorinner which is full of holes is to be attached to an outer panel bywelding. One robot holds the outer, one holds the inner. The robotgrabbing for the inner would use a camera system located on its endeffector (or near its end effector) and guide the robot in using certainholes on the inner panel as the targets. The CAD system program would bedesigned in such a way as to allow a three dimensional simulation of therobot's movement to be made, and to determine the robot cycle time whichwould be achievable in the accuracies of position using those targets.This would be done by working through the known accuracies of the camerasystem, and the photogrammetric equations could also be utilized (whichwould be called upon from memory) and the whole system designed on theCAD system. The other robot picking up the door outer would work in asimilar way, and other points on the door outer that could be picked upwould be monitored and the total assembly predicted using the controlequations of the two robots.

Each operation involving a particular part that might go through a linecould then be tracked. For example, now that the inner and outer trackare assembled, the assembly could also be sensed from the holes on theinner or other features in the operation of putting the door onto thecar. This too could be automatically tried out on the computer.

The key here is that the target type solution allows us to quantifyeverything. Every single possibility can be predicted by mathematicalequations which can be put into a simulation program used to try outdesigns.

One key advantage obviously is that if the whole operation is not fastenough or accurate enough, other types of targets might be specifiedthan certain naturally occurring ones. One would then know that inplanning the line layout for this operation, and one might have to usetwo robots per operation instead of one, etc. All of this could be knownin advance.

Also, the CAD system could design the way in which the differentstations interact on a line. For example, certain targets might becovered up by assembly operations and alternate targets would have to beused. Indeed, complete alternate strategies in case certain targets weremissing could also be programmed into the CAD system and the wholedesign checked out so that such targets were present in the initial CADdesign part. The assembly operations usually require the build up ofcomponents on targeted tooling which is used to hold the portions inplace while other components are added. If such tooling is targeted, thelocation and precision of the robots, etc. doing the assembly can bevastly reduced.

We should consider the problem of assembly with and without toolingplates. In one case, where the robot holds the part to be assembled, theother robot assembles another part to it having a target. Perhaps eventhe first robot's gripper is targeted, which in essence then is aprogrammable tooling plate. In fact, changeable grippers can be used, ascan changeable tooling plates, all with targets to facilitate theirchanging.

Continuing the ideas of these RPS things, it is not just usable ontooling fixtures for assembly. It is also welding or any other kind offixtured operation where one would put the object onto the tooling andthen add another object to it. Any sort of holding fixture will do.Again, another robot (if it is still enough to act as the fixture on theother side) can be used assuming the part is targeted.

Also covered in this case is the concept of covering up targets on anobject when the object is assembled to it. For example, a radio knobattached to a targeted radio front plate with targets around the knobpost would be suitable. Also suitable is the assembly of a tail lightbezel to a car with targets around the attachment points.

For example, if everything could be put together from the targets in astatic location, this would be most advantageous. Also desirable is toonly move the part or whatever simply to 2 or 3 such static locations.For example, right now instrument panels are put together on acontinuous line where they go around a continuous loop with maybe 16,20, or 25 people standing in the loop assembling all clusters etc. intothe instrument panel.

Assuming that the problem of the wiring could be solved in an instrumentpanel line, one might replace this with a single fixed panel. FIG. 7shows a fixed assembly cell 900 for an instrument panel, engine or othercomponent 902 into which would be placed all of the various items byone, two, robots or more robots 904 movable on tracks 910 going out andgrabbing them from different containers 916 located nearby. There arecertain advantages to cell 900, namely that one does not have to dealwith a moving line and that everything is fixed (which means the highestplacement accuracy that could be expected.) In this condition, one wouldhave to be able to pick out all the different parts. There might be 20to 25 parts required for the panel, plus the multiplicity of differentversions of the same thing, which might lead to a total of 100 differentchoices with the robot. This would require either very well positionedparts or a good vision system, and the targets would make it quite easyto do both fast and accurately.

Speed and accuracy would be paramount because if this system could notgo fast, it cannot possibly compete with humans on a moving line.

Most of the parts of the instrument panel or engine 902 as shown in FIG.7 could have targets on them, which could be made to seem like theexisting parts of the panel or engine 902 (which have lettering on themor certain points or corners and holes). These are all already presentin abundance on a panel or engine 902, for example. In addition, knobson the panel can be used to cover up targets and other whole assembliescan cover up the targets on the inside of the panel which would be usedto guide it in.

The central problem with this then may not be the targets at all but thetooling. If a universal robot is utilized, the robot might have to havedifferent end effector tooling and thereby lose cycle time changing it.Such a robotic work cell is thus very appealing. However, parts wouldhave to be brought in when they ran out from outside the periphery ofthe cell. In other words, what is essentially parts trains 906 wouldcome in on tracks 908 or 914, just like tracks into a roundhouse.

With a car body, clearly the body in the first place is made up of alarge number of panels welded together. This is done in-line, and ateach station a varying piece is brought in, fixtured and the car bodywelded up. This could conceivably be done without the fixtures usingtargets.

For example, suppose we wish to build a car body up from scratch. Wecould take stampings that comprise the side of the car, e.g., startfirst with the cowl and the A-pillar and assume that the cowl is weldedto some sub-frame. The A-pillar could be brought in using targets on thecowl and targets on the A-pillar. The A-pillar would be accuratelypicked up and placed by the robot on the cowl where it would then bewelded once it was in place. The accuracy of the positioning would betaken from the targets on the car, not from a fixture.

Satisfactory target accuracy could be achieved with an A-pillar roughly2 ft. long welded to a body cowl. This piece should be accuratelypositioned within+/−10 thousandths. Whether it is, in practice, isdebatable however. First, one would figure out just what the targeterror is to do this. In any case, with one robot holding the part perthe camera unit (which would now be not on the robot at all but off tothe side so that it can get the true reference of the A-pillar to thecowl), another robot comes in and welds it. This can be done withdynamic correction, if necessary, by the first robot for position due todistortion by the welding robot. This could be called “fabricationmethod with dynamic real time target location”.

After putting on the A-pillar, one could then go over and put on theother A-pillar, so that one could come in the body side. Indeed, all ofthe parts could be welded on this way, particularly those internal ones.This is really just like a completely flexible fixture. With differentprograms for different parts, one could make anything.

Using targeted outer parts, you can then drill all of the holes neededfrom the targets etc. Again, one would have to bring all the parts toit, so that it would then move the finished subassembly out of this celland bring another one in. Instead of having a continuous line, you wouldhave a relatively few number of stations, relatively widely separated.This would allow, however, different things to be made more easilybecause on a continuous line, the varying part steps would not match up(although the same technology could apply to that too).

To do an assembly like this, you might have multiple cameras too, oneabove the whole cell 900 and various ones off to the side coming in fromevery angle. This would allow you to get a better fix on each of theparts as well as deal with the fact that sometimes you would obscureyourself with the tooling doing the operation.

An interesting thing is a co-target, that is a target that tells wherethe other targets are. Such a co-target identifies the part and as suchtells where the targets are. It also gives the orientation coordinatesso that one can find the other targets.

The beauty of this one market idea is that the single market identifiesthe part and therefore all the targets around it. It could also give anapproximate orientation code for the part, which now could include notjust a reticle (which was shown in the turbine case noted above) butactual dot code or something from which an accurate orientation in 6degrees could be created. From that, one can then find the otherfeatures of the part that one wishes to use for further use such as thecomers, the holes, etc. Therefore, there is a one-step process usingthat single marker to tell the rest, including where the others arewhich in turn are then used.

Other kinds of targets include: a serial plate, either just as it is,specially designed, or with print markings; a logo; a bolt head (asmentioned above); and an instruction plate or a bar code marker.

Getting targets on sheet metal can be a problem as lots of sheet metalreally does not have any holes in it per se, at least on certainsurfaces that one might want to use. In such a case, we can painttargets on those for the assembly operation noted. Marks such as smallindentations, ridges, tabs or the like can also be stamped in.

Painting could be done in a stamping fixture coming right out of thepress, since the part is trapped anyway usually upside down. It wouldnot take too much to lay it in another fixture and determine targetlocations. Another possibility would be to actually stamp the target in.This would seem to be impossible on an outer surface unless one stampedon as part of the stamping process something that would essentially comeoff.

The other obvious place would be in a weld fixture where one clearlywelds most panels together with some other panel. Obviously, you cannotreplace the weld fixture with a robot and targets if one does this, butat least the assembly of that welded up component to any other could betarget automated. Weld fixtures typically also have sensing associatedwith them to create an in-line check fixture situation. Even this checkfixture could actually contain the marking units.

Obviously there are numerous underbody and chassis components that couldbe targeted without any detrimental aesthetic effects - mufflers, tailpipes, brackets, clamps, tie rods, steering parts, etc. These areobviously labor prone things to assemble and one could envision robotsworking underneath the car while the other work is going on above in theassembly function. This is, of course, very tiring work for people andtherefore would be well set up to robots. There is no reason why thewhole operation could not be up above the floor. The car is brought inoverhead with some robots on a second tier and other ones underneath.

This system should be usable in a variety of circumstances, such as:components that are all assembled to the body in the trim area includingtail lights, glass, front, rear, wheels, headlights, grill, doorhandles, rear view mirrors, antennas; internal assembly components suchas door assembly, lift mechanisms, trim, and instrument assembly coveredabove; and underhood assembly (this too probably can be marked up in anyway desired with no detriment).

Note that another type of marking, purely decorative, is trim decalswhich could be judiciously chosen as targets.

So far all target sensing development is based on a raster scan camera.These, therefore, were generating windows, tracking, etc. Other types ofspecial cameras might be interesting for this application. For example,if one always has 4 targets you could have literally 4 image chips with4 separate scans of the field which would then simplify the processing.In addition, one really does not need all the number of lines that thereare in a field if one uses targets that occupy large portions of thefield. Therefore, a camera having 10 vertical lines and 10 horizontallines widely spaced might be quite suitable. Instead of scanning say 300lines with a camera, you would then scan only 20, but this wouldpresumably be sufficient. This would require a target design that wasable to be cut at multiple points and give the right answer. Naturally,the speed of response of this would be quite a bit higher.

A polar scan swept like a radar screen is also possible. Although theseare not widely available, one could assume some future value.

One of the beauties of putting targets on the A-pillar, etc. is that thefixture can actually slip, and dynamic correction be made via the highspeed of response to the target position.

One would also expect that you would have to have 3 or 4 targets to goalong with all of this or else you could not do it. The other thing isthat you would certainly want to have redundant targets in case some ofthem were knocked off or the paint gun did not work, or whatever.

One can also come up with a plan for automating the way in which theparts are brought in. Targeted little trucks would be provided thatinstead of having fixed tracks can actually reposition themselves atdifferent locations, either in the horizontal plane or even vertically,to be grabbed by the robot. This ability to have them at multipledifferent planes with a very high reaching robot and targets also allowsfloor space to be conserved. Indeed, another robot could be used to loadthe parts into the racks from the targets out of the dunnage. In thatcase, therefore, the parts would be presented to the assembly robot in abetter orientation while still targeted, possibly saving some assemblytime. The other unload robot then would be used to essentially deal withthe messy conditions.

From the above, it appears that all things can be made from targets. Ifrobots can accurately position and are structurally stiff enough towithstand the forces of welding, drilling, or whatever (and this is notquite sure), then presumably all things can be made this way. However,some things apparently cannot. For example, if one has to back up adrill, this can be done with a robot in some cases, but not in othercases.

In a structure such as this, important considerations would be: wherethe cameras would be located, what kind of robots would be utilized,would it be better to have more than one robot holding each part orclusters of parts, etc. Particularly, the concentration should be on thetarget orientations and the cameras or the like.

As noted above, the following are parts of the car in which, the theory,targets would mean very little in terms of their add-on aesthetic value:under hood, under carriage, when covered up by something else as inseats over bottom part, inside door covered by trim, inside of taillight covered by light, grill covering other part, etc., inside trunkonly to a degree. Those are the main areas, but they may constitute atleast half of all the assembly work, if not considerably more (in fact,much more).

The other thing to be considered is can the body structure be built upwith targets and then the targets taken off. This seems logical. Everyweld fixture of every panel can be a place for target application.

Incidentally, there are people talking about painting the whole carafter it is assembled, not just sticking painted doors on. That would ofcourse make the job even easier. Then everything is targeted going inand you just paint over them.

If parts are targeted, then obviously, an RPS equipped robot can pick upa part in a very well known way. Assuming that the gripper is correctfor this, it can then (within the accuracy of the robot itself, whichalso could be RPS controlled) present this part to another robot in awell positioned manner. Ostensibly, the accuracy of this position bejust as accurate as if it would be fixtured, assuming that the fixturingdetails and the RPS were equal in accuracy. The only problem would bethe inherent inaccuracy of the robot.

However, it should be noted there is nothing to keep the second robotused to assemble something onto it from using targets as well, thereforetaking out that positional error. Conversely, the targets utilized bythe second robot could be on the first robot as on its gripper. In otherwords, if the first robot grabs the part in a known way, the secondrobot only needs to home in on the first one, not on the part. Thiscould make it a lot simpler for an assembly operation. A very clear-cuttarget board could be utilized on the gripper or somewhere on the end ofarm toolings that would be directly relatable in a known way.

This two robot assembly is very similar to the human using his twohands, one to hold the first part and one to hold the second, and indeeda twin armed robot could be considered for the job as well. It is morearms and hands than it is robots per se, although they have to beseparately controllable.

It is however interesting to note that the inherent accuracy of theposition is considerably higher than for a human. The human, if thereare no marks on part at all, would have a great deal of difficulty forexample, drilling three precise holes exactly positioned in a location.The human himself would need some kind of fixture as in a template orsomething. Wherein in this case, the targets actually provide that sincethey are carried on the part. They can be on there in a way that a humanwould not be able to directly relate to in an easy manner. Theadvantages over humans are noted.

There is a special assembly technique that lends itself to this. Oneclear-cut aspect is targeting parts in unobtrusive zones, as on the backside away from the observer. For example, a radio knob on which the backof the knob had the targets or a door handle of a car in which the innerdoor panel was targeted (this is almost a natural case anyhow since itis full of holes which can be used as targets). If we then use one robotto grab at the inner side where the targets are present, we can thenassemble the outer side which might be a visible one onto something.

The outer is usually the part you would like to grip so that you couldassemble it to something else like the shaft on a knob or the body on acar. This is not easy to do if the targets are on the inner. What ismore straightforward, therefore, is to use the targets on an item thatit is to be assembled onto as in around the shaft or around where thedoor goes (the opening).

One possible way to do this is to have ‘U’ shaped tooling where it isgrabbed on the rear part but held on the front via the ‘U’.

One thing is sure, you could take doors that way and use drilled trimthings on the outside doing it that way. This is if it all lined up. Wemight have to do this on the finished car, however, targeting theoutside.

One could also grab the part from its targets and then move it over to adrill fixture rather than a second robot. The fixed drill fixture orwhatever it is could be different for different cars say-this would ofcourse give it flexibility. The robot would take the part and move it toone of five stations depending on which type it was sensed to be, andthen orient itself relative to that station which itself would betargeted.

In the latter example, it is a gang drilling, gang welding or some othermulti-position gang operation with multiple points. This would be trueof a robot grabbing parts up willy nilly and just putting them in theright thing. Note however, we are talking about non-fixtured parts inthis case. Parts that are not being dropped into a fixture, onlypresented to a gang drill that itself does not have any fixture locatorsbecause the targets have located it.

However, this might not work since in some operations the robot couldnot hold steady enough and you would have to drop it onto locators. Butin other cases, it would just present it to it. The interesting thingabout a robot, like a human, is that it could just go to one drill andsequentially drill four holes in a row using the targets to set up thestraight line or whatever the other pattern was. This could be totallyvariant depending on what part it was, without the necessity of buildingthe special gang drilling fixtures. This could be the putting ofmoldings on, but really it could be anything (even cylinder heads orsomething else) if one could position accurately enough and hold it, ofcourse.

One obvious thing that you do not have to position ultra accurately isspot weld guns. This would be relatively simple. You could sequentially,using targets, put a part set to a spot welding gun that would then putdifferent spot welds depending on what it was (or glue or anything elselike that). Therefore, one robot which would target on the gun that itcould aim the part off of, could actually do numerous parts in manyways, all programmably. However, such a robot would obviously not bevery fast as it is a one at a time deal. This would be the oppositeeffect of having gangs of spot weld guns on fixed automation coming inon a part that is transferred through. In this case, a single robotwould go make all the welds on a part, and it would be transferred out.It would take many more robots, but that might not be more expensive Youget to optimize the one gun you do have, or any other things such asadhesive sealer or guns or whatever.

The same logic holds for presenting a part that has numerous threadedholes in it to an automatic tapping machine or an automatic nut runneror for that matter taking a nut runner and moving it to different pointson a part.

One interesting question is whether all this allows whole other assemblyprocedures. If you do not need special tooling and fixtures, you couldjust put a car together by feeding parts to the assembly point such ascell 900 depicted in FIG. 7 just as if a human was going to stand thereand build a car in his garage. He would go out and get the parts andbring them in and assemble them up. The advantage of this is that youare moving the parts by conveyors to the build points, rather than thecars to separate build points. The robots 904 are going out getting theparts and coming back. Robots, for example, could be mounted on slides910 or for that matter could be gantries themselves. This would looklike a round house or something with each of the slides then being fed.However, you still have to get the parts in and out of the system or thefinished cars have to be moved as by conveyors 912 and 918.

It does mean that the car as it is built up is fixed in a known place.Thus, each place you do something is dimensionally known from theoperation before. This is not true in line type assembly unlesselaborate shot pining is taking place. Therefore, the total build up ofthe car is a known commodity even if the positions of the parts and theother things feeding into it are unknown.

In this new mode, each of the parts of the car is targeted and as theyare laid upon each other the assembly itself becomes targeted insofar asthey are not covering up the old pieces. This means that as each robotapproaches it can memorize the locations or use the targets. Indeed, thetargets on some of the parts as they are positioned offer a chance toupdate the robot coordinate system so that it can re-memorize thepositions in case the robot gets out of sync.

For welding, a robot would bring the parts in from one side, lay themdown and the second robot would grab arc welding tooling and weld themall. Such a method that could be totally programmed in a CAD system.This completely relieves having to transfer data to fixture builders,etc., shaking down lines and so on since one can theoretically do thisin CAD and you are dealing with minimum numbers of pieces of hardwareand delivery time therefore. One robot, perhaps two robots would beworking on a single area, each with different sets of tooling that theycan use. Different robots might pick up parts from different sides withan overhead gantry type robot as well.

This whole procedure would just keep proceeding until the car was built.Obviously there cannot be much room for parts washers in this type ofthing. Instead, the car would be transferred out on conveyors 912 and918 after the body was welded up to a second location where it would bewashed and then transferred in to another similar type station forassembly. In this case, all parts would be brought in from the outsideand the assembly built up. One might do this in two stages, assemblingthe inner parts of the car and then using the assembly robots to goahead and weld the top on. This type of procedure allows one tocompletely change ones mind as to how you are making the car. Make itone way one day and another way the next. You do not have an investmentin the lines and fixtures. This is really a big advantage.

In order to achieve 60 jobs an hour, you would need 60 such robot cellseach putting a car together in an hour for example. A big question ishow would you feed the parts to the cells. Note that 60 jobs an hour isa typical number for both body plants and assembly plants but someJapanese run 30. We can take that to be the minimum.

Clearly some sort of very clever material handling would be required inorder to get the parts to these individual drops. The simplest of all,of course, is an overhead monorail conveyor 914 bringing the parts inwith the robot simply going over just as the human does in atransmission plant and grabbing off the one it wants in place for itsjob. This would be by far the cheapest, but would require numerousconveyor loops which might get in the way of each other since we aretalking about multiple types of parts coming in at once. The loops mighthave to go in a circle around the whole operation with the robot movingradially outward from its job to find the part it wants, grabbing it,off and bringing it back.

A second way is to simply have them in racks 916, bring the racks in (asmany of the parts are racked anyway) and bring them in with trucks toeach of the individual job sites. For example, if each rack contains 16parts, we can do a day's production with a single rack full at any onecell. The true flexibility of this is a distinct advantage.

One of the fascinating aspects of all of this is that probably theoptimum way in which the car was built could be totally laid out on aCAD system. Different cars would use different sequences or differentideas. You could build anything given this approach, just as humanscould in a garage if they were presented with a kit of parts.

Note too that the CAD system could also design the target on thepart-specifying special stamped in (cut in, molded in, etc.) targetsetc. at certain points on the part surface. The CAD system too would notonly know the location of the targets to other part features, but to allother targets on all other parts used to form the assembly.

With reference to FIG. 8, an item 1010 is shown being moved in thedirection of arrow A below an overhead conveyor system 1011 generallycomprising a track 1012 on which a carriage 1013 is movable. Object 1010is suspended in any suitable manner from a hook 1014 secured to carriage1013. Hook 1014 includes an aperture 1015 for a bolt 1016 for securinghook 1014 to carriage 1013. A fiber optic 1017 is embedded within hook1017 and has a light receiving end 1018 in a surface 1019 of hook 1014and a light emitting end 1020 in a surface 1021 of hook 1014. Fiberoptic 1017 can be any fiber optic element such as a single fiber opticor a bundle thereof, of which several are commercially available. Aplastic ‘corfon’ fiber optic element is quite suitable.

In FIG. 8, it is assumed that it is desired to determine the position ofitem 1010 suspended from hook 1011 when item 1010 is at a generallocation designated 1022, in FIG. 8. It is also assumed in FIG. 8 thatitem 1010 is in a known position relative to the position of hook 1014.

A light source 1023 is positioned above track 1012 to direct lightdownwardly such that it will be incident on the upper surface 1019 ofhook 1014 of carriage 1013 when a carriage is positioned below the lightsource.

A light detector 1024, in this case a scanning matrix photo detectorcamera comprising a lens and a detector array comprised by a pluralityof horizontal rows of discrete photo diodes, is positioned adjacent theconveyor so as to be adjacent hook 1014 when conveyed item 1010 islocated generally in position 1022. More particularly, when the conveyeditem is in position 1022, the light emitting end 1020 of fiber optic1017 is imaged by the camera lens 1040 onto the matrix array whichprovides real time information as to the location of hook 1024.

In a typical case, illustrated in FIG. 8, the light 1025 emitted fromend 1020 is imaged to form a spot 1025 on four adjacent photo diodes ofarray 1024. As the photo diode array is scanned, an output signal 1026,indicative of the position of the spot of light 1025 on array 1024 isconveyed to suitable means 1027, such as microcomputer, to determine theposition of hook 1014, and thus item 1010, relative to any knownposition, such as the position of a robot, or the position of detectorarray 1024. A signal 1028, indicative of the position of hook 1014and/or item 1010 is then conveyed to suitable robot control means forcontrol of a robot, not shown, for manipulation of the hook 1014 ofconveyed item 1010. A signal 1029 controlling the robot thus includespositional information concerning the hook 1010 or item 1014 to bemanipulated by the robot.

It will be readily apparent that the matrix array in FIG. 8 providespositional information in the x and y directions in the plane of thedrawing. It is also possible to readily provide information concerningposition in the z axis. For example, as shown in dotted lines in FIGS. 8and 9, a further fiber optic element 1030 may be embedded in hook 1014,extending from horizontal upper surface 1019 to front surface 1031 whichextends vertically and transverse to the plane of the drawing.

Thus, a further linear photodetector array, positioned to detect lightemitted from light emitting end 1032 of fiber optic 1030 would provide asignal indicative of the position of hook 1014, and thus item 1010 ifdesired, in the “z” axis, that is, in a direction transverse to theplane of the drawing. This signal would be processed in the same manneras signal 1026 to provide a three-dimensional determination of therelative position of the conveyor hook and/or suspended item such as itsposition relative to a robot. Alternative means utilizing additionaltargets for providing three dimensional data as to hook location usingmultiple targets are disclosed below.

In the embodiment depicted in FIGS. 8 and 9, the elongate lightconducting means is shown embedded in hook 1014. In some instances, aswhere hook 1014 is a monolithic cast metal item, it may be moreconvenient to fix the fiber optic to the hook in some other way as bysimply gluing or otherwise adhering to an outer surface thereof. In thatevent, however, it is preferred to provide a housing for the fiber opticto prevent damage in use. This is readily achieved by providing a grooveor slot in a surface of the hook, in which the elongate light conductingmember can be laid and thereafter covered with a protective material,preferably opaque.

Where the invention is utilized to determine the position of a member oflike objects, such as the position of a plurality of identical hooks1014, it is preferred that the position of the light emitting end orends of the elongate light conducting member are in substantiallyidentical position on each item. Where this is not practical, or wheremore precise position determination are required, the position of thelight emitting areas of each object may be calibrated.

In the embodiment of FIG. 8, the light conducting members are fixed tohook 1014. It will be readily apparent, however, that the members befixed to suspended item 1010. In the latter event, position of suspendeditem 1010 is determined directly whereas, in the latter case, it isdetermined indirectly by determining the position of hook 1010 and byknowing the position of a suspended item relative to hook 1010.

FIGS. 10A and 10B illustrate a basic application of the invention to animportant sector of robot usage, that of taking parts off or placingparts into continuously conveyed containers or transport media. There issubstantial amounts of labor worldwide utilized in this materialhandling procedure. In addition, many assembly operations require ahuman to first take a part, for example, off a conveyor, (for example,an overhead monorail conveyor here illustrated) and assemble it to someother part. He may have to then put the part back on such a conveyor. Inother words, only if the conveyor interaction problem can be solved, canthe assembly process be automated.

A particular embodiment of the invention is shown here utilized toremove transmission clutch parts off an overhead monorail carrier. Inthis particular carrier, 1200, there are two parts, 1201 and 1202,resting in a pocket of the carrier. To keep the cost low, the carriersare typically made out of angle iron, bent rod etc. and are not overlyprecise in any direction. In addition, they are conveyed often from anoverhead rail 205 and can swing in the direction of motion, side to sideand twist over limited angles. They can vary easily in their positionfrom the reference point of the monorail +/−½ inch and as time goes on,they degrade still further due to repairs, substitutions etc.

Rather than attempt to build highly precise conveyors, it is of extremeinterest to provide a robot system that can deal with this particulartype of conveyor, not only allowing one to retrofit existing plants, butfurther keeping future conveyor costs low-at the price of certainadditional sophistication in the robot hardware. As can be seen fromthis particular example, however, the embodiment of the inventionprovides a system which can be made at low cost, much less than that ofproviding precision conveyors capable of being used with robots withoutthe invention.

As shown in the top view, a robot 1210 is positioned to grab the partoff of this particular monorail choosing one of the two parts in thisparticular carrier. At a later time, it may choose any one of parts suchas shafts 1220 to 1223 located in another carrier 1225 on the samemonorail, to be used in assembling with the first parts. For example,robot 1350 is used to assemble the parts pulled off the conveyor byrobot 1210.

The robot 1210 can be of any particular type. However, a Cartesiancoordinate robot is generally preferred and a specialized one for thispurpose is shown in the drawing. It is noted, however, that polarcoordinate robots can be utilized, although they require much morecontrol sophistication. A polar coordinate robot on moving linear slidesparallel to the conveyor can also be utilized but requires added cost.

As shown, cartesian coordinate robot 1210 has an arm 1231 moving in andout toward the conveyor line 1205 and it moves along an x axis slide1232 parallel to the conveyor. The third axis is the vertical axis or zaxis out of the plane of the drawing 1234.

In this invention, the carrier has been provided in this example withfour targets located, in this example, at the four corners of thecarrier, 1240 to 1243. These targets can be any type, for example thosedescribed in this application. The use of four such targets is notnecessary, but is preferable in many cases to provide a full six axissolution of object position with redundancy. Three targets is sufficientto provide the solution alone. The carrier also contains, desirably inthis case, a snubber rail 1245 located beneath the carrier which cancontact certain mechanical guides such as 1250 and 1251 to restrain theside to side motion. These guides are also shown in the end view (FIG.11). They can optionally be spring loaded as shown such as 1251 to keepthe snubber up against the stationary guide 1250. A lead-in on thesnubber guides is shown in the top view.

FIGS. 10A, 10B, and 11 illustrated targets such as the four as shownaffixed to the carrier. In this case, these targets provide a signalfrom which the sensing camera unit such as 1260 can lock on. This cameraunit can be located either on the robot arm as 1261 or external to therobot as 1260 (shown mounted by the side but also mounted above thestation). If it is external to the robot, it may also be desirable toalso have targets such as flashing LEDs, 1265 on the end of the robotarm which can also be tracked to provide relative data. Both types 1260and 1261 can be used to provide data from different points of view.

When the robot sensor such as 1260 has locked onto the carrier (or otheritem, see subsequent embodiment) it can then track this conveyor even inits side to side motions in a reliable manner due to the very highsignal to noise ratios of the targets as will be discussed below. Thisis vastly superior to looking at the parts or carriers from their graylevel images in this kind of swinging and uncertain environment. It isnoted that the referenced Pinkney invention or other photogrammetricsolutions can offer high resolution data in up to 6 coordinates, x, y,z, roll, pitch and yaw. This is fully sufficient and often more thansufficient to accomplish the job, particularly if constraints such asthe snubber guide rails 1250/11251 shown are utilized to restrain themotion in one or more axes.

Once the main camera unit such as 1260 has locked onto the conveyor andcan compute its position for the feeding to the robot command computer1280, then a second system (such as 1261), either using a differentcamera or simply different lighting, circuits etc. can tell where thepart is in the carrier.

An important feature of the invention is the use of one camera unit totrack the targeted conveyor (or other object) while a second subsystem,even with the same camera, senses the part within the conveyor carrieror on it. Such a second camera or subsystem is described in referenceno. 6, and can provide up to five axes of data itself (x, y, range,pitch and yaw). This system can be right in the gripper as in FIG. 8 ofreferenced copending application (Ser. No. 200,401).

It is noted that the sensing of where the part is in the carrier doesnot necessarily have to be made at the robot station. It could be madeupstream, using for example sensor camera 1262. This serves theadditional purpose of signaling the robot system if any out ofspecification situations exist, so as to abort the attempt to grab thepart. This could be a badly mangled carrier, a carrier with no parts atall, a carrier with the wrong part, etc. Thus, identification of thepart obviously can be done as well as sensing its location on thecarrier, pallet, or whatever.

Once the decision has been made as to where the part is and the factthat it's a correct part, the robot moves in to remove it or converselyto place another part back on the carrier. In this case, the controlcomputer 1280 of the robot takes the data as to the coordinate positionof the targeted carrier and continually updates the robot's information.With a Cartesian coordinate axis, it is particularly easy to make thisapproach since one can simply run parallel to the conveyor (x direction)and only take out the differences in position relative to the parallelline of motion. While this can be done with a polar coordinate system,it is much more difficult to do dynamically. In any case, one needfollow the carrier only in an approximate sensor using the gripper, andpossibly other sensors to take up the difference.

The tracking target approach, for example, using hardware such asdisclosed in Pinkney or a twin stereo approach with two cameras, can beaccomplished using a camera tracking both the gripper and the carrierconveyor (or part) and/or with a camera mounted on the robot arm itself.The reason why this is so successful is that it tracks targets which canhave, and maintain, high visibilities even in an industrial environment.These targets can be differentiated by means of intensity, color orshape. Any and all can be utilized by this system.

For example, in the end view of FIG. 11, light 1300 from light source1301 behind the carrier can be provided which illuminates the targets1240-1243 at the four corners (or any other place). These targets can besimple apertures in plates such as circles, squares, triangles,etc.-whatever is distinctive and can be discerned by a computer visioncamera such as 1260 or 1261. A triangle aperture 1302 in target plate1241 is shown for illustration.

Alternatively, the targets can be comprised of color filters such as1305 and indeed a different color could be used for each differenttarget (1240-1243) or different part carriers if desired toautomatically code which is which, if this is a problem, as it could bein certain more random applications. In this case, when utilized withwhite light source 1300, the color of the target is an immediateindicator. However, in some industrial environments, maintaining acolored filter may be harder than a simple slot.

It is noted that when utilized as shown, with the light source behind,it may be desirable to put a diffuser such as ground glass 1310 (dottedlines) in the slot or near the slot (but not necessarily on the carrier)such that the light is directed over a range of directions. Other moredirectional diffusers such as diffraction gratings, prismatic devicesand the like can also be used where more light is desired at certainangular locations such as the approach path of the robot, or in thedirection of camera 1260 for example.

It is also, of course, possible to use the fiber optic based targetssuch as disclosed in FIGS. 8 and 9 above and in U.S Ser. No. 200,401.

A final type of target of use on a system such as this is aretro-reflective target, such as plastic prismatic retro-reflectors,retro-reflective tape, rectilinear material and the like. This is shownas target 1242 in FIG. 11. In this case, a light field 1320 must beprovided to illuminate this. If reflectors having a high degree ofretro-reflective capability are utilized, the light source should becoming from the same angle as the sensor (e.g. from 1260). The sourcecould either be fixed or mounted on the robot. The reflected light fieldis shown 1321 directed back along the incident path.

Let us now consider the question of targeting the parts themselves or acontainer of parts such as carton 1579 travelling on a pallet inconveyor of FIG. 17 wherein the targets are simply printed on.

There are many means of implementing such targets on parts, althoughthis is obviously somewhat more difficult since one has to consider thefunction of the part and often it's aesthetics as well. From the pointof view of the robotic system, however, targets need to be such that atleast three of the four targets for example are visible in order toprovide a satisfactory 6 axis solution to the photogrammetric equations.Under certain circumstances, where there are more constraints, perhapsonly one or two targets need be visible.

In continuing the example of FIGS. 10A,10B, and 11, it is noted that itis sometimes desirable to have an auxiliary robot such as 1350 (possiblywith polar coordinates as shown) to take parts such as 1202′ which robot1210 has pulled out of the carriers and loaded onto an assembly fixture1351. This robot then assembles different parts such as the shafts 1220′for example that have also been pulled off and puts the two parts 1202′and 1220′ together and then shoves the completed assembly down on achute onto an outfeed conveyor 1360.

Alternatively, robot 1210 can be utilized also to perform the assemblyoperation particularly if it is provided with rotation about the y axis.A dual robot system, however, is faster since one can be assemblingwhile the other retrieves more parts.

The converse is also true, the previous assembly can be going on whilerobot 1210 puts the assembly back on a monorail conveyor of the sametype. For example, in this particular application, a second conveyor canbe located right under the first conveyor 1205 on which the assembledparts were placed. This conveyor could be floor mounted or overhead.Robot 1210 could also turn around 1800 and put a completed assembly on aconveyor parallel to 1205.

FIG. 12A

FIG. 12A illustrates a car door targeted according to the invention 1352travelling on a conveyor 1353. As it passes a light source 1355, fourfiber ends 1359 are simultaneously illuminated and light emanates fromthe opposite fiber ends 1360 to 1363 which then form the targets. Thesefiber ends are flush with the door panel which itself may be plasticsuch that the plastic fiber blends with the plastic door. Indeed, thefibers carrying the light 1370 to 1373 may be cast right into theplastic in the injection mold. They may be in the door panel sheet, laidin just as if they were regular glass fibers in a SMC (fiberglass) door,or maybe carried in or adjacent the ribs of the door (if any).Alternatively, where there's an inner and outer panel, the fibers may beplaced in between.

A camera unit 1368 looks at the light from the fiber ends 1360-1363.This camera may be located overhead the conveyor and/or on a robot armcoming in to pick the door up for example.

The light source itself may be pulsed to create a higher signal to noiseratio of the targets relative to ambient light. It is noted that each ofthe fibers comprising the targets may transmit colors to allow colordiscrimination on that score. Indeed, one might even think of the fibers1370 to 1373 as bundles of fibers. Indeed different numbers orarrangements of fibers and different arrangements of the target ends1360 to 1363 could be used such that varying codes were used todelineate which was which and what door type etc.

It is noted that paint, ink or other film or coating type targets can besprayed on parts utilizing spray marking guns. Particularly effectivefor this in terms of producing nicely shaped targets such as trianglesand other items that have a very recognizable shape is the DiffractoTurboJet, U.S. Pat. No. 4,269,874. Clearly, a marking station on thepanel for example could be utilized to spray certain targets on withpaint. These targets can be provided with special paint such asphosfluorescent, infra red absorbing or ultra violet reflecting,something that would tend to distinguish them under certain types oflighting. They could even be clear coating which would be invisible tothe eye but under certain lightings would fluores or absorb certainwavelengths preferentially. The latter would be particularly nice forfinished parts. However, any sort of paint could be utilized for exampleon an unpainted door which would later be painted just as long as it waschemically compatible or would be removed in a normal preparationprocess.

FIG. 12B illustrates the color filter plate 1380 which can be placed infront of the group 1359 of four fiber ends of FIG. 12A. This colorfilter plate has filters red, yellow, green, blue as shown which causethe light emanating from each of the fiber ends 1360-1363 to show thosecolors for example. These colors could all be infra red or any othercolors.

An alternative for differentiating which target is which, is to actuallyuse different light sources for each of the fibers and modulate them atdifferent rates or pulse light on each of the fibers sequentially.

FIG. 13 illustrates another application of the invention, this one beingto a windshield 1400 the edges of which as well as other features areviewed by cameras 1405 and 1406 having an included angle, _between themsuch that ‘stereo’ vision in terms of depth perception is provided.

This stereo vision or any other vision of this part is made vastlyeasier by the delineation of the object edge provided by the ‘lossy’fiber 1411 which runs around the periphery of the part emanating or‘leaking’ light at each point as it goes when illuminated at its end bylight source 1410.

While the whole edge has been delineated in this particular example, itis obvious that only sections of the edges of a part such as this arerequired for accurate placement of it. It should be noted as well thatit allows for actual mensuration of the part itself since the contour ofthe windshield edge is desired to insure that it will fit into thewindshield opening of the car in a correct manner.

The fiber in this case can be cast into the windshield glass at the timeof manufacture and could even be a glass fiber itself simply of slightlydifferent characteristics. Alternatively, this fiber could be reallyjust a portion of the same glass made in such a way as to convey lightaround the windshield periphery.

It's obvious that these same principles could be utilized on plastic orfor that matter, even on metal parts where the fiber would simply beglued onto the periphery of the part or be covered over only in part bythe metal if desired.

It should be noted again that the light emanating from the fiber can beinfra red or any other wavelength that might be desirable for betterdelineation of the surface. Indeed, the fiber in the case of plastic,can be buried such that it could not normally be seen under visiblelight but that infra red radiation, in this case emanating from thefiber outward through the glass or plastic could be seen.

Clearly, more than the four targets shown in FIG. 12A on a door can beused as can more zones than simply the periphery of the part of thewindshield such as FIG. 13. However, these are the two principleexamples, namely four points is the sufficient solution including aredundant check of the six axis, photogrammetric equations, and ofcourse the periphery is the main item of interest when looking withstereo cameras.

Note that in FIG. 13, many fibers are lossy just by themselves. Ifadditional losses were required, for example when they were imbedded ina matrix, the fibers could be roughened for example.

Clearly, plastic pallets and carriers can also be so instrumented.Indeed, light conductive paths can even be built into the plastic of acarrier such as FIG. 11.

FIGS. 14A and 14B illustrates another method for targeting objects. Inthis embodiment of the invention, a cylinder head 1500 is targeted, inthis case on the rocker cover rail 1501, with targets 1502, 1505, 1506and 1507 on the four corners of the rail perimeter.

Target 1502 in this case is formed by a single depression in the rail,for example, the conical surface made by a drill as it just touches themetal. Such a conical surface reflects light at diverse angles from thatof the flat machined face of the cover rail itself, and therefore makesan excellent contrast target when viewed at appropriate angles. Forexample, when light from source 1510 carried on robot 1512 hits thecover rail containing such conical target, light is directed back atcamera 1511 from the target because the face of the target is pointedmore or less at the camera 1511. However, the light from the cover railis directed off at a angle away from the camera. Therefore, it appearsbright against the background of the cover rail and against that of thecast surface of the part. In other cases, the rail surface would bebright and the conical surface dark. The fact it is a cone, means thatapproach from any direction in the plane of the rail face desirablyproduces similar results.

Shown in the top view for illustration, different sets of targets havebeen shown on each of the corners. However, it is considered likely thatin any one case, one type of target would be used. For example, thetarget cluster 1505 contains four such conical faces or for that matter,four targets of any sort such as will be shown in FIG. 15 for example.In this case, four targets of course are much more unusual than a singlepoint and would be unmistakable relative to any sort of backgroundssince nothing else on the part could have such a cluster reflection. Inthis case, the center of the four dots gives the center of the target.

The same holds true of 1506 which is a three pointed version, alsoproviding a center. Cluster 1507 having two points, while probablyunmistakable, does not have a center point except in the one plane. Inthis case, therefore, the center of the dots themselves would have toprovide the answer in one plane.

FIGS. 15A and 15C illustrates another target method, also applied inthis case to cylinder heads but of course general to any sort of part.In this particular case, the part is cast where it is shown that oncylinder head casting 1530, there are appendages cast which do notinterfere with either the assembly or the function of the part. Theseare shown as 1531, 1532, 1533 and 1534. These appendages are indeedtargets and of course are unmistakable in any sort of view.

To make them more unmistakable, certain angles have been cast into theirsides. Such as shown in the end view of 1532, the particular angle ofreflection transverse to the head axis is such that when overhead lightfrom an overhead light field 1535 is projected, that these facets shootlight off to a camera at a preferred angle. In this particular case, theopposite one 1531, would have to be made in a different way as shown,such that it too had a facet in that direction. It is noted that thedirection can be chosen such that no other features on the part havereflective angles in the same direction. In other words, for any part,no matter what it is, one should be able to find certain angles at whichtarget data can be made to show up either brighter or darker than therest of the part with no other, or at least a minimum of other partfeatures having angles at these directions. This of course helps thediscrimination in a passive manner.

Similarly, certain targets can be beveled in more than one plane, suchas 1533 as shown such that when viewed from either of two angles, abrighter reflection is shown. (Conversely the lighting can be at anangle and the camera located overhead.)

Also shown in this drawing, are cast in target cones or crosses such aswere applied in a separate drilling operation in the FIGS. 14A and 14Bversion above. These are shown as 1540, 1541, 1542 and 1543.

In this case, a male portion in the mold itself provides a suitableindentation in the part. Since the targets can be so cast, they can beof many other shapes besides simply conical surfaces, holes or othereasily machined shapes. For example, cross shapes like the facets of aPhillips screw, which are undeniably discriminate as targets as opposedto other features on the object which can have some resemblance toconical shapes.

It should be noted as shown, that such shapes do not necessarily have tobe indented in the part but can be raised such as the equivalent feature1545 shown sticking up. Such a knob or bump on the part, however, can bein the way of the function of the part and its handling if it is notproperly positioned. It is therefore thought since many parts generallyhave flat surfaces, which are either functional or to simplify handling,that the best means is an indentation in those surfaces which does notinterfere with either purpose.

As shown in 16B, a cast or drilled in cone, cross etc. in a surface 1545into the surface of material 1546, can optionally have a transparentplastic filler material such as 1550 placed into it to cover up aportion or all of the depression or even provide a raised portionsticking out from the surface of 1546.

This plastic material can serve several purposes. One purpose is that itsimply protects the surface of the depression from rust anddeterioration. This could be quite important in let us say a brightshiny drilled portion on an aluminum or steel part which in timetarnishes or rusts.

A second purpose is that the filler itself makes a different opticalelement out of the mirror formed by the cone surface (cross etc.) and inthis case forms it into a prism which can have use for either spreadingthe light or directionalizing it.

A third potential reason is that the plastic filler may itself be chosenso as to preferentially reflect light only of certain colors. This thenallows another form of discrimination of the targets based on color.

The principal disadvantage of using such a filler is that a separateoperation must be made to put the plastic in, which cannot normally bedone easily on a machining line. One exception, however, is the processwhereby first the reflective hole is drilled into the casting, thenplastic is sprayed into the holes very simply, and then a finalmachining pass required for other purposes is done which in the processshaves the excess material away leaving the plastic flush with the holesurface. This obviously then only adds the spray guns to the process.

FIG. 16 illustrates another example of targeting, this time on a plasticbody panel 1570 illuminated by light field 1571 which is then directedby a target molded into the plastic surface 1572 onto camera 1575. Asshown, the target is reflective and is composed of a diffraction gratingwhich directs particular colors or, in general, light of all colors, atangles from the surface. An alternative is that the target 1572 becomposed of a multi layer interference elements preferably in plasticwhich also can direct light at preferential angles in preferential colorcombinations.

If the camera 1575 is capable, as is a color TV camera, of color sensingas well as spot shape sensing, it can then differentiate these colorsand unmistakably identify that such a color spread can come only fromsuch a target. This can be done even in the presence of strongbackground, as from the surface of object 1570. Such color combinationscan also be coded into the targets to identify the part 1570, its angleof orientation etc.

Rather than mold the plastic into the part, it can also be simply gluedonto the surface of the part (1578). If a thin reflector film, such as1578, it may, even though sticking up, be non-interfering with thefunction of the part. However, for plastic outer body panels in cars,the flush target mounting such as 1572 is preferred. These targets onpainting of the car, become covered over. For example, if the targetsare on the door panels of the car, which are mounted to the car at thetime of painting, their presence is lost once the car is painted. Thetargets used for such mounting for example, should be flush and createno disturbance with the panel surface once they are painted. It is alsonoted that targets can be built into objects however to actually be partof the object's appearance. The necessity of covering the target updepends greatly on the aesthetic characteristics of the object.

Another possibility is to utilize targets which are seen as targets onlyunder special illumination which would not normally be present in ahuman situation. For example, consider target 1572 which could be eithermolded into the panel or for that matter, simply a portion of theplastic surface of the panel itself treated with a special ultra violetflorescent material. Only under ultra violet light would this targetportion of the panel actually be visible relative to its surroundings.

This is partly true of the multi layer diffraction case above where theline spacing of the diffraction pattern or the multi layer material andspacing could be chosen such that only under certain colors ofillumination, and then perhaps only at certain angles, could the lightbe strongly seen relative to the surroundings. This then would beparticularly easy to arrange if such wavelengths were in the UV or infrared just beyond human vision. The near-IR is an excellent region forsensing with present day solid state cameras for example.

It should be noted that such targets do not necessarily have to bemolded in but could be evaporated onto the surface such that the raisedamount of material is virtually negligible. Naturally, in crudeapplications, such applied targets such as 1578 could simply be whitecrosses of plastic glued on. Clearly, this could be unobjectionable inthe final product if in the final painting process there is a wash thatsimply removes the glue and target. One advantage of the fiber typesshown in FIG. 12A, is that the fiber end can be extremely bright andflush to the part surface.

FIG. 17 illustrates another application of the invention, to tracking acarton such as 1579 travelling on a ‘car’ or carrier conveyor such as1590. The carton has been randomly placed onto the carrier and it isdesired using programmable robotic means to grab the carton and pull itoff at a certain station.

To accomplish this, all sides of the carton have printed on target setsof which 1580, 1581 and 1582 are visible in the drawing. These targetsets can be of any usable description and remain with the carton always.The beauty of this is that they can be utilized for tracking in themanufacturing plant, and for robotic warehousing purposes and throughoutthe distribution chain, even for example in a supermarket providerobotic unpacking of the product and place the product on the shelves.Naturally the product packages within the carton, such as let us say eggcartons, or milk cartons, cans etc. can also be targeted for the samepurposes since all those have printed on labels or the like.

While each face in this case is shown with four dot type targets, thesecould clearly be of any number or type. The carton also could be codedto indicate the goods within the carton. In the extreme case, this wouldrequire a UPC type codes (e.g. 1591), and indeed a miniature UPC codeitself could constitute one or more targets. However, it is consideredthat most of these would use much less complicated codes since therewould normally be no need for such large amounts of information.

It may well be necessary to code the object or targets since alldifferent boxes would have different target spacings due to their ownshape and one would first wish to decode which type it was so that thespacing of the targets could be known to the computer of the robotmechanism and fed into the calculations for the various solution of thephotogrammetric equations.

For example, it could well be that a special code target might also beused such as 1591 which would include all the photogrammetric solutiondata for that carton plus an indication of what was inside if desired.The robot camera system could read the code first and from samedetermine the various target location data on each of the faces of thecarton including the target shape and size, the target spacing, how manytargets there were and for example the shape of the product itself,whether in a square carton or what have you.

FIG. 18

FIG. 18 illustrates a similar concept this time using targeted toolssuch as the grinder 1600 driven pneumatically via air hose 1601. It isdesired that a robot with a gripper come pick this grinder up and dowork on an object, for example the leaded-in zones of a car body at thepanel junctions.

For this purpose, the tool gripping area 1610 itself is targeted, inthis case using light emitting diode targets 1605, 1606, 1607 and 1608,also fed through the umbilical 1601. These diodes can either be oncontinuously, may be flashed to provide has high signal to noise, orrippled such that only one is on at any given time. The latter is usefulif photo sensors responsive only to one point at a time are used such ascontinuous spot detectors (e.g. UDT SC-10).

Naturally, rather than light emitting diodes at the tool, fibers can beutilized to feed this data to the same points from one or more remotelight sources.

The robot hand with camera would approach this tool and via the targetsgrab the tool at the desired location, in this case, the cylindricalsurface 1610 which would be grabbed by ‘V’ shaped grippers. It is notedthat the targets can be placed specifically so they bracket this areaand this is a preferred mode of target placement in such instances.However, this is not necessarily required. The targets could be placedsuch that gripping would be known to the computer to occur in any otherlocation as well.

In this case, it may also be desirable to have a code such as 1620shown. This code could carry with it the data of where the part is to begripped, whether it's between the targets or somewhere else, and again,what tool it was and perhaps other data as well.

It is noted that a target such as 1532 and 1531 of FIGS. 15A and 15C, ifthey project from the object in one or more planes, can allow moreaccurate solutions of the various pitch and yaw data which are derivedfrom the projected spacing of such targets viewed by the camera.However, the more the target projects from the part in question however,the more the possibility it is objectionable for handling or aestheticreasons.

FIG. 19

FIG. 19 illustrates another application of the invention to the assemblyof car bodies. In this case, it is desired to assemble a deck lid 1660onto the body opening formed by two fenders 1661 and 1662 and the otherportions of the body not shown for clarity. This problem is very similarto that of fitting the doors in a door opening or the hood in the hoodopening and is optimally improved using optical sensing as disclosed.

As shown, a robot arm 1650 carries with it a tooling fixture 1651containing vacuum cup fixture 1652 and 1653 which attach to the deck lid1660 to be put on. The fixture itself contains optical sensors, in thiscase 1670 and 1671 which are used in the mode shown here trackingtargets as well as to measure certain variables of the part itself usingconcepts shown in the referenced applications.

Such applicable sensors are shown in reference nos. 6, 16, and others.

As the robot approaches the car body containing the fenders carryingwith it the deck lid 1660, sensors 1670 and 1671, which contain linearor matrix camera units, have determined the position of the deck lidrelative to the cameras themselves. In other words, the cups 1652 and1653 can pick up this deck lid in a relatively random fashion from letus say a roller conveyor and have the cameras compensate the robot forthis random position by sensing the edges of the deck lid.Alternatively, the sensors can sense the deck lid edges ideally andcause the pickup to be made in the correction location. It is likely toothat other cameras would be located on the other sides of the part, forexample, as shown as 1680 (dotted lines).

When the robot is relatively far away from the body, the camera unitwhich also contains for example illumination source 1675, picks up thereflected image of a stamped in cone targets such as 1665 into thefender. Alternatively, for even better contrast, a hole 1665 can beprovided, back illuminated if possible by a light source such as 1666.Unfortunately, however, in most portions of panels, extraneous holes arenot desired. Such stamped in targets are however extremely possible andcan be accomplished just as in the case of the cast in targets of FIG.15A and 15C in an analogous manner. The fiber based targeting systemsare ideal if they can be employed economically.

As the sensing of targets such as 1665, and 1667 on the opposite side,as well as other targets around the rest of the periphery of the deckopening, allow the robot system to home in on the body. Note that unlikeprevious embodiments, it is not a single camera which is seeing alltargets, but the ensemble of two or more cameras whose combined targetdata gives the position and orientation to the part. As the camerasensor unit comes in for its final approach, an oblique light projectorunit such as 1672 illuminates the portion of the part itself from whicha triangulation data as to the exact range to the fender 1661 can beprovided at a higher resolution. Such a sensor unit incorporating thishas been shown in reference no. 6 and other references.

As the part then fits into the opening ‘D’, the gap width ‘W’ on eachside is sensed by each of the cameras on the four sides and optimizedfor the car body in question. When the deck is optimally positioned,then various hinge screws and bolts are run down to lock it into place.This process therefore not only generates a fully automated deck (ordoor) placement, but also creates a optimal body fit for highest qualityperformance. This operation does not necessarily require the use of thetargets and can be done in a targetless fashion particularly if the bodyis stopped when this is occurring. If the body is however in motion, thetarget data definitely is very much desirable such that its side to sideand forward/backward oscillations can be tracked on the approach.

It may also be required that two sets of camera magnifications be used,one at high magnification to determine the distance ‘w’ and one at lowermagnification to track the targets. This depends on the application andnaturally is not as desirable as just a single unit. Further, in thiscase the targets are shown being covered up by the panel, in otherwords, they are out of sight in terms of the body itself. This can betrue in both doors, decks and so on. Some of the tracking however couldbe done by targets which were visible on other portions of the body andnot covered. This would allow the targets to be tracked even at the timeof actual panel insertion and bolting which would be desirable on movingparts. For this purpose, it is thought special targets should be stuckonto the body such as target 1681 shown which has been stuck onto thefender and is, for example, comprised by a white background with a crosson it. Such targets might be viewed by a completely separate camerasystem mounted to the side or overhead or on the robot arm 1650 itselfrather than on the tooling.

It should be noted that in any of the above embodiments, targets shouldbe as distinct as possible. If possible, certain types of reflectivetarget material such as plastic retro reflectors and retro reflectivetapes can be of extreme interest as long as they can be placed on theobject in a manner that does not ruin its function. Such tapes andtargets, therefore, are best suited for use on objects which do not havean aesthetic purpose and some of these would certainly be all conveyorparts, cartons etc. The problem, however, with these targets is thatthey are generally of materials which must be attached and this cancause difficulties in terms of both the cost of attaching the targets inan accurate manner (remembering that for best operation, with multitarget systems, the target spacing and orientation needs to be known,such that the photogrammetric calculations can be accurately solved. Thesecond problem with these materials is that they are often plastic andin some cases, plastic will not survive the remainder of the processwhether it be hot washes, heat treat, or what have you.

It should also be noted that targets, when applied can be removed foruse on subsequent parts. For example, retro reflective glass targets ofvery high contrast can be screwed into tapped holes on the part at verywell known locations and screwed off later on and used again. This wouldbe easily accomplished for example, on the cylinder head of FIGS. 14Aand 14B if the tapped rocker cover rail holes for the rocker cover wereutilized to carry the targets which were screwed into those holes,preferably automatically. At the final rocker cover installation, thesescrews would be taken out and the rocker cover bolts put in. Naturally,however, this adds two operations, screwing in, and screwing out, to theprocess but it does utilize the same holes that are put into the partanyway. Other targets could be attached with glues etc. which could betaken off the part with solvent and off the target such that it could bereused again after cleaning. This is discussed relative to FIG. 20.

In addition to the above ideas, there are several other continuationsfrom the previous application that should be noticed. For example, FIG.8 of U.S. Ser. No. 200,401 discloses instrumented grippers with fiberoptic sensor units including a triangulation projection source to whichallows three axes of data to be obtained. It is noted herein that up tofive axes of such data can be obtained using projection of multiplebeams or four beams to get four or five axes of data. This allows thepitch and the yaw of the part to be obtained as well as the range, plusthe xy image and is further described relative to FIG. 22 below.

It is noted that the robot arm may be instrumented for guidance withsuch a sensor either using LED or diode lasers as targets or via fibers.Such concepts of guiding robots with external cameras using targetedarms has been shown in the copending application of reference no. 17.

Color discrimination of the various targets can be made by using colorTV cameras or with simply a color sensor in front of a detector ifapplicable. For example, relative to background levels, if all targetsare infra red emitting such as infra red LEDs, then an infra red, bandpass filter can be placed in front of the camera such that greatlydiscriminates against the white light image background and showsprimarily the infra red targets superposed thereon.

Furthermore, the holes put onto cylinder head in FIGS. 14A and 14B canbe more than just conical, they can be actually be drilled in deepersuch that they tend to absorb all light. In this case, one would look atthe angle of reflection from the bright machined face of the rockercover rail and the target holes would show dark.

It should be noted in FIG. 15B, a blob of plastic or a blob of siliconecould be put on top of the part to act as a target. Indeed, if a linearstrip of silicone for example were utilized, this would approximate thefiber arrangements shown in FIG. 12A or 13 and indeed light can betransmitted through and around the part illuminating edges of itthereby.

FIG. 20

FIG. 20 illustrates one example of a reusable target, in this case, aspecial screw 1700 which is screwed into a threaded hole 1701 in a part1702 such as the cylinder head of FIG. 15, engine block, or for thatmatter just about any machined part that has a threaded hole. Thesethreaded holes would as has been pointed out, be almost certainly holesthat already exist on the part for other purposes and as for assemblywith the target part taking the place of the regular part up until thepoint of final assembly when it would be removed.

The target screw is built like a socketed cap screw but instead of thesocket hole, in this case, being at least partly filled with aretroreflective target 1705, which is ideally comprised of plastic orglass retroreflective material for example that commonly used onautomotive reflectors or specialized types built by 3M and othercompanies.

If desired, a color filter such as 1710 can be utilized on top of thisscrew or as part of the retroreflector to give a preferential colorsignal from this particular bolt or stud if it is desired to distinguishit against others. The reflector design itself may also provide suchdistinction being multi-pointed or what have you.

This particular arrangement provides an extremely high targetdelineation, and allows the targets to stand outward from the partsurface if desired (as for better photogrammetric solution purposes) bysimply having a long threaded length. Furthermore, this stud is arelatively low cost item and can use automatic lines to put in and takeout. The only disadvantage of course is that it must go into a hole thatis later used which means in the final assembly process, the targetcannot be used unless the part is not moved during assembly after thetarget is taken out.

While a screw type has been shown, it is clear that other arrangementssuch as bayonet, snap in/snap out, or other targets could be utilizedwhich could be removed with special tools from otherwise clear holeswhich later would accept trim strips, rivets or what have you.

In other cases, the target itself might simply have a pointed end suchas a pin which could be stuck into the object material and later removedleaving a hole which would cover itself over if the material wasrelatively compliant. This could include, for example, seat materials ormeat on overhead conveyor lines where the carcass itself could havetargets put in it.

FIG. 21

FIG. 21 illustrates an application of the invention to working on acontinuously moving car body assembly 1780. In this case, a roboticsystem according to the invention is provided complete with camerasystem 1785 which locks onto the body targeted with reflective targets1781-1784 in the working region causing the robot to track the motion ofthe car body side to side, backward and forward on the body “truck” (notshown).

The sensor unit 1785, in conjunction with robot control computer 1789,controls the robot arm 1800 to move an abrasive belt grinder 1801 togrind out the lead 1790 fill-in between the sail panel 1791 and the roofpanel 1792. There are two forms of additional optical sensor units ofuse in this embodiment. The first is 1805, such as FIG. 16 of thereferenced application which allows the attitude of the belt grinder tothe surface of the body to be determined for tracking purposes. Thesecond (not shown) is a contouring sensor such as FIG. 4F of Ref. No. 16which contours the leaded zone of the body to feed back contourcoordinates to the grinder and update the amount of metal left on andjudge whether or not further grinding should occur and if so, from whatangle (determined in conjunction with the dynamic tracking data at lowresolution from the target sensor 1785, and at high resolution from theon-board sensor 1805).

Utilizing all three of the optical separate sensor systems plus forcefeedback, a complete grinding cell so to speak can operable on-the-fly.If the car can be stopped in its motion, the target based system is notas much required for tracking the gross motions of the body and theother two sensor systems are sufficient. However, the target system is agood “insurance” for rapid approach.

In the above application, considerable amounts of specialized hardwareare of use, much of which has been discussed in the referencedapplications. For example, camera units are best provided by solid statematrix arrays such as the GE TN2500 and the new solid state TV colorarrays now appearing on the market by Sony and others.

In terms of light sources, flashed Xenon light sources are very good forilluminating targets with brilliant high signal to noise pulses, evenwhen color filters are applied. Also, such flashes do not cause thesolid state cameras to bloom, a desirable advantage.

Desirable laser light sources include diode lasers operating in theinfra red made by RCA and Laser Diode Laboratories. Of interest too isthe Mitsubishi 4001 laser diode which is partially visible.

The high powered infra-red LEDs such as the Texas Instrument types canalso be utilized for such illumination through fibers or what have you.LEDs are very convenient in that they are low power consumption and canbe modulated as can the current range of diode lasers.

The approach described relative to FIG. 12A and 13 holds for all kindsof other parts such as tires, parts of aircraft, furniture, just aboutany part where some sort of method of casting, molding or otherwiseplacing fibers into the part can be done. Even metal parts can haveintegral fibers if they can stand the melting temperature (e.g. quartzfibers).

It should also be noted that the part does not necessarily have to havefibers cast or molded in. One can also have a fiber placed onto thispart, for example, glued to the part around its periphery or at specificpoints. These are then illuminated and can then be used for the samerobotic and other purposes as shown above.

This gluing operation, however, generally requires additional labor,either human or robotic, although it could be done on an automaticin-line machine as well.

It should be noted that while fiber optics have been discussed as thelight carrying medium, it is clear that a transparent silicon bead laiddown on a part is also light transmitting although less so. Thisparticular use of fibers and other light transmitting mediums appliedinto or onto parts is particularly appealing for many applications wherethey are to be substantially robotically handled and, therefore, wherethe cost of applying the fibers in and illuminating them at differentstations is made up by savings due to reduced complexity of roboticautomation utilized.

The application of such concepts to things such as tooling was discussedin my recent copending application (ref no. 6) on robotic castinginspection, where sensors were in the tools to sense part condition.This disclosure has expanded on this to provide fiber illumination oftool location to allow handling or size determination of tools. This isalso related to a copending application entitled “Method and Apparatusfor Detecting Wear or Breakage in Tools”. Suffice it to say that toolscan also be illuminated like the J-hook of FIG. 8, to provide meaningfulindicators or targets to allow pick up by robots or other automation.One can consider such tools as cutting tools, small drills, routers,pneumatic wrenches, saws, lasers, weld heads etc. All can beinstrumented in this manner. Even small things such as sockets forwrenches can be so instrumented.

Note that ‘light’ in this application refers to all wavelengths ofelectromagnetic radiation IR through UV.

Similar fiber optic emitter targets can be the grippers or arm robotsthemselves, replacing LEDs or other types on the grippers such as shownin copending application entitled “Electro-Optical System for Control ofRobots, Manipulator Arms and Coordinate Measurement Machines”.

Suitable fibers include, at the low end of cost, the Dupont Corfonplastic fibers as well as glass fibers made by American Optical, Corningand numerous other manufacturers.

It should be noted that image transmissive bundles can be utilized toremote the images of sensors shown in this application as has been shownin the referenced copending application which this application is acontinuation in part. Such fiber optic bundles are made by Nippon SheetGlass, Olympus and others and can have very high resolution.

It is noted that image scanning photo detector camera arrays and solidstate TV (matrix array) cameras, while preferred, are not the only meansof viewing the targets of this invention. Other TV cameras can be used,as can in some incidences scanning laser beams or even fixed detectorsoptimized for a preferred target signature. Continuous or quadrantposition detectors (such as UDT SC-10's) can be used as well todetermine the image position of a single spot or target at a time.

Shown in FIG. 22 is a sensor according to the invention providing animprovement on some of the fiber optically based sensors of theco-pending application U.S. Ser. No. 200,401. This particular sensorshown is a multi range sensor of unique small size according to theinvention which in this case is shown being so small that it can bebuilt into the grippers of robots. It does not require targeted objects,but can be combined with other embodiments to work in conjunction withtargets as well.

As shown sensor 1900, located in this case in one half of the gripperportion 1901 of a robot end effector is comprised of light sources 1905,1906 and 1907. (In this example there are three light sources althoughthere could be any number.) These light sources are diode lasers or inmany desirable instances, they are 0.005 optical fibers remotelyconnected to diode lasers with only the fibers brought to the sensor.

In any case, light from each of the fibers is focussed by single lens1910. However, due to the variation of positioning of the fibers, thelight is focussed at different distances and at different anglesdepending on the position of the fiber. This is ideal for providing amulti range, multi resolution sensor, with highest included angle andresolution at the shortest ranges as is desired for accurate partpick-up and other purposes.

Light source 1906 is focussed at the nominal range to the part 1911shown in the drawing forming a reflected spot 1912. This spot is, as hasbeen described in many copending applications, imaged by lens 1915 ontoan integral photo detector array 1916 (dotted lines). However, in thiscase, again for compactness, the image is formed onto a coherent fiberoptic bundle 1918 and carried to a remoted matrix photo diode array.Thus, in this example, all light sources and sensing can be done overfibers if desired. This is attractive for thermal and electricalisolation purposes, plus light weight on small robots.

A suitable window, 1920 is provided in front of the sensor housing.

The other two light sources, 1905 and 1907, on either side of thenominal, focus at different distances and at different angles. Thelarger the included angle theta, the more the resolution. Therefore, itcan be seen as the image forming capability associated with 1907 is atthe highest resolution with the part closest and this is used for thefine approach of the sensor where the range ‘H’ might be only half aninch. In the case of 1905, ‘H’ might be set up for 10 inches.

Obviously, this sort of an arrangement is fine for maintaining areasonable focus of light sources at different ranges. However, with asingle lens 1915 one needs a narrow aperture to give large depth offield and maintain the spots projected in reasonable focus over a widerange of object locations. Alternatively, a zoom lens 1915 can be usedto maintain focus over the range.

Since spot centroids are being measured, it is noted the spot image canbe somewhat out of focus and still be utilized (see reference no. 6 orno. 8 for suitable circuit processing). Optional white light sources canalso be used with this arrangement to provide a edge image lighting withthe part 1911.

It is further noted relative to FIG. 22 that each diode laser or fibercould be focused by an individual lens. While more complicated, thisallows more angular spread between beams. It is contemplated that onlyone beam would be turned on at once, suitable for the range in question.However, even if more than one were on simultaneously only one isgenerally in the field of view of lens 1910 at a time. If there are twoin the field, they can be discerned from their location. Indeed, twodivergent beams can be projected on purpose at once in the field, one todetermine range and the other to give angular orientation from the beamseparation on the target, knowing range.

It is further noted that this invention is very useful to controlrobotically positioned non-contact processes such as laser welding,drilling etc. especially on continuous lines. In terms of processes ingeneral, the invention applies to welding, drilling, grinding, cutting,hardening, and any other material removal, addition on transformationprocess.

The characteristics of targets used in this invention generally includedistinctive shape, light reflection, light transmission or lightemission characteristics relative to the normal surface of the objecttargeted. Where the ‘normal’ object has targets, a better definition isrelative to the rest of the object surface, i.e. the untargetedremaining portion. Light emission, reflection or transmission can bedistinctive in color, direction, distribution of direction or color,shape, and intensity.

In the case of the fiber version and other active targets, the targetscan also be diverse in their light modulation frequency.

It is noted in the application of the invention to practical plantproblems, that photodetector arrays are much preferred over the analogtube based TV cameras used by Pinkney and other photogrammatists.Particularly photodiode arrays such as the GE TV2500 do not requirefrequent calibration and therefore can be relied on much more to giveaccurate dimensional data as to target or spot location. For example, aTV tube drift of 3% in the apparatus of Pinkney et al can create agenerally intolerable error of 0.3″ at 10″ standoff in the range dataalone. The arrays used in this invention preclude this possibility.

It is further noted that in the embodiments shown herein relative tocontinuous conveyors, if conveyor speed is known, the trackingrequirements are reduced accordingly.

It is also noted that the snubber rails 1250 and 1251 are but oneexample of means to constrain motion or velocity in one or more axes ofan object in this invention. It may also be useful to constrain velocityfor example using electro magnetic or viscous fluid damping. Constraintsof this sort generally make the total robotic handling or parts workingsystem easier to control.

This disclosure has described many ways of adding targets to objects.Other ways of making the target part of the object have also beendescribed. Where the object is one which is in it final form and locatedin a position that it can be seen by a consumer who expects it toprovide a pleasing appearance, there is considerable requirement to makethe targets used in the invention either essentially invisible oralternatively make them have aesthetic value of their own.

For example, a doped target zone of a plastic dashboard piece canfluoresce under UV light but remain invisible in normal illumination.

Alternatively, a portion of the object may contain a special dopant tocause it to reflect or absorb in the IR more than normal.

A desirable condition exists if one can make the targets part of theoverall design to provide for example a pleasing accent feature whichare viewed as part of the design. Where possible the targets can also befunctional features such as holes, knobs etc.

For multi-target application involving three, four, or more targets (toprovide maximum solution capability of the photogrammetric equations) itis noted that the targets do not have to be equi distant or otherwise orrigidly arrayed in their relation. Thus a variety of aestheticpossibilities exist. For example:

On rectangular parts, the four corners are desirable where targets canbe squares, circles or other shapes.

On circular parts or sections thereof, the four targets 90° aredesirable or three targets 120°.

On irregular parts, the targets can be in any logical arrangement.

Examples are:

a furniture leg 1950 with round head tacks 1951-1955 at corners servingas targets (FIG. 23A)

automobile grills 1960 with decorative square target fascets 1961-1965or slots near the corners (FIG. 23B)

automobile steering wheels 1970 with triangular bright inserts 1971-1973in outer edge of three 120° spokes (FIG. 23C)

It is also noted that within the target slots 1241 etc. of FIG. 11,transmissive diffraction gratings and other preferential diffusers oftransmitted light can be located.

It is further noted that light from the fiber end 1020 for example neednot necessarily be imaged by lens 1040, but can be detected directly byone or more detectors.

It is noted that in many plant applications where the targeted objectgoes through many processes or are repetitively used (e.g. the conveyorcarriers of FIG. 10), that the targets can degrade or be knocked off,destroyed etc. Thus it is desirable in many cases to have a superfluousnumber of targets.

Since only three targets are needed for a complete photogrammetric sixaxis solution (and even two will do if certain aforementionedconstraints are used), two or three is then the base number of targetsper object generally desirable. In many cases, a fourth target isdesirable to provide a redundant solution however.

This invention therefore considers the purposeful addition of extratargets also in known locations relative to the two to four basictargets and for the additional step of determining which targets arepresent and using an optimal solution for those.

Consider FIG. 24. Illustrated is a conveyor pallet 1980 used repeatedlyin a plant equipped with targets 1981-1984 and extra redundant targets1986-1988. In one mode of operation, normally camera 1990 and computer1991 are programmed top consider only targets 1981-1985. If, however,one of these targets is missing, the images of target-1986, 1987 or 1988is utilized. In general the rationale is to use the remaining targetclosest to the missing one but the best rationale is to use whatevertarget combination gives the best solution (i.e. most accurate) of thephotogrammetric equations.

It is contemplated that certain additional targets might also be appliedto provide, in certain instances, more accurate solutions for example,to pitch or yaw variables in the plane perpendicular to the lens axis.For example, one might choose at certain stations in the line where ahigher degree of sensing in one or two of the variables were desired, touse target 1988 on purpose, instead of 1985 say.

Naturally, if all targets but three are damaged, one uses the remainingthree regardless. However, the invention can include the additional stepof signaling a control that pallet SN 1368 say is down to its last fourtargets and should be repaired.

It is further noted that a system input or verification station is oftendesirable in a system such as shown in FIG. 10.

For example, consider providing sensor 1262 at a position where theconveyor carrier is well positioned such that the targets can be checkedfor presence and their locations verified if desired. Not only is thishelpful in keeping the system in control, but if each carrier wasserialized or sequenced the actual target location can be measured atthis station and the locations stored in a computer, such as 1280,relative to the carrier in question. This allows different carriers ofdifferent parts in different states of repair to be intermixed on thesame line with no loss of target position accuracy. This is importantsince the accuracy of the solution of photogrammetric equations (used toguide the robots on the line such as at the station of FIG. 10) is basedon the degree of accuracy with which the relative location of thetargets to themselves and to the carrier body is known.

Only one such verification station (which could also have a carrierserial code reader as could the station of FIG. 10) is required perline. It also, as has been mentioned, helps monitor damaged carriers anddamaged carriers could then be automatically routed off-line for repair.

Note that verification station can also be used for parts when they arein a fixtured or otherwise known correct position at some point in aline. Any missing targets as well as locations can be verified beforethey enter the system.

This invention will be useful in a wide range of applications and it isenvisioned that a standard sensor computer unit can be built essentiallyindependent of the application. The user would only need type in orotherwise enter data into the computer such as 1280, to tell the systemthe pertinent target location and spacings on the parts or objects to behandled, assembled, or worked. Thus it can be reprogrammed to differentparts, lines etc. and forms the basis of a generalized robot controlsystem.

What is claimed is:
 1. A method of successively determining a 3D position or orientation of an object using a detected set of targets on said object, said method comprising: providing an object having a set of original targets which are detectable by an electro-optical sensor, the number of original targets in a viewed portion of the object exceeding a number of targets required to determine the 3D position or 3D orientation of said object; viewing a portion of said object with an electro-optical sensing means having an output; analyzing said output of said electro-optical sensing means to detect, from those original targets of the set which are detected, a first subset of detectable targets in the viewed portion of the object; further analyzing said output of said electro-optical sensing means to determine the 3D position or 3D orientation of said object using said first subset of detectable targets; subsequently viewing the portion of said object with said electro-optical sensing means, where at least one of the detectable targets of said first subset of detectable targets in the viewed portion of the object is missing, obscured or degraded; analyzing said output of said electro-optical sensing means to detect, from those original targets of the set which are now detected, a second subset of detectable targets in the viewed portion of the object which excludes at least one detectable target of said first subset which is now missing, obscured or degraded; and further analyzing said output of said electro-optical sensing means to determine the 3D position or 3D orientation of said object using said second subset of detectable targets.
 2. A method according to claim 1, wherein said analyzing to detect steps include a selecting of the subset of detectable targets based on a determination of a detected quality of those original targets of the set which are detected.
 3. A method according to claim 1, wherein said analyzing to detect steps include a selecting of the subset of detectable targets based upon a determination of which targets would provide the most accurate solution to a photogrammetric equation.
 4. A method according to claim 1, wherein said subsets of detected targets each comprise three targets.
 5. A method according to claim 1, wherein at least one target in said set of original targets comprises au LED, and wherein said method further comprises illuminating said LED.
 6. A method according to claim 1, wherein at least one target in said set of original targets comprises a fiber optic element, and wherein said method further comprises illuminating said fiber optic element.
 7. A method according to claim 1, wherein at least one target in said set of original targets comprises a retroreflective target, and wherein said method further comprises illuminating said retroreflective target.
 8. A method according to claim 1, wherein said electro-optical sensing means comprises at least one TV camera.
 9. A method according to claim 1, wherein said electro-optical sensing means comprises two TV cameras positioned to view the object in a stereo relationship.
 10. A method according to claim 1, wherein said analyzing to detect steps include a selecting of the subset of detected targets using a microcomputer.
 11. A method according to claim 1, wherein said further analyzing to determine steps include the utilizing of a data base of said object in determining said position or orientation of said object.
 12. A method according to claim 11, wherein said data base comprises a design data base of said object.
 13. A method according to claim 1, wherein at least one of said original targets is a natural object feature.
 14. A method according to claim 1, wherein said first subset includes all of the set of original targets. 